I'm not familiar with the experience of shooting 4x5 handheld, even though I do have an old Graflex 4x5. But I'm not sure that MF is any slower to use than 35mm, other than having to change film more often. In particular I'm thinking of my MF rangefinder systems like the Mamiya 6 and the Fuji GF670. These are very compact, agile, easy to use and of course the IQ blows 35mm away.
But I guess the bottom line is, you use what you want to use. Of course if the bulk of your photography is professional work then factors like practicality, efficiency and overhead may come into play. Fortunately for me it is purely a hobby!