I'm not an LFer but will offer an opinion. You've pretty much stated it yourself. HP5 gives you the two stop speed advantage, but FP4 gives you finer grain. Having said that, for a 400 ISO film, HP5 has reasonably fine grain. I'd say use both. When and if you need the speed, use the HP5, and when you have a "slower" subject and more time, use the FP4.