Welllllllllllll, if you start off by pulling ISO 400 TX to Tri-X to 250, you're knocking off 1.5 stops right there and still only gaining 1.5 stops over ISO 100. So in broad daylight as you mentioned, even using the sunny 16 rule, I'd say you'd be working at f-8 to start with without the ND filters.
When you go through all this with Tri-X, I think you're just obviating the need for the faster ISO and the trade-offs of using it with the grain structure that you get with slower ISOs like Ilford Pan-F or Fuji Acros. But some like that kind of grain. Personally I try to avoid it which is why when I pull TXP I process it or have it processed in T-Max RS.

And without being critical, I'm trying to understand why you want to shoot portraits at f4 anyway. I say that because generally, every lens has a sweet focus spot, usually somewhere between f8 to f11. So if you're working at f4 to soften the background, you may not be getting the sharpest image of your subject. There are many ways to play with camera placement and/or lens choice to manipulate depth of field aside from shooting near wide open in order to soften the background. But you know that, right? Why not use 5.6? How much background softening are you looking for?

I'm also thinking if you're using a ND to knock off four stops shooting a portrait, you may have trouble trying to manually focus the image although at your calculation, probably just a 1.5 or 2 stop ND pulling Tri-X to 250 would suit the result you're trying to get or maybe just moving the person into open shade would eliminate the need for the ND filter altogether.

What I'm saying is that I prefer to match the film to the purpose and unless it's impossible for me to move either my camera, the subject or both for background purposes, I much prefer shoot portraits to use the sharpest focus of the lens I've selected. I also don't like putting any additional glass in front of my lenses if I can avoid it.

But if you like the results, afterall, that's what really counts. It still comes down to personal preference.
Take it light ;>)