as usual, David is completely right about everything!
Yes, there will be an obvious difference in an 8x10 print produced from 100 speed film vs. 400 speed film. On the other hand, subject movement and poor depth of field from handheld work with 100 film will look worse than the increased grain of 400 film.
Personally, I have trouble handholding 35mm without using an EI of at least 200. In medium format handheld I often shoot Delta 3200 rated at EI 800 developed in full strength Xtol for 9 min at 68. Very liberating!