I've wonder this myself over the the years, and here is my thought process.
1) Standard lens for 35mm format is 43mm, for MF it is 85mm. That is essentially one stop when it comes to hand holding. Moderate telephotos become impractical.
2) Shutter/mirror vibration is going to be greater in MF, further reducing the hand holding speed. (But I shoot OM's, which colors my opinion of acceptable noise and vibration.)
3) The overall kit (lenses, body, etc) is going to be be bigger and heavier. (See note above about OM's.)

The solution? Folders!!