I have carried a 4x5 on a mountain bike for a few thousand miles, so it is possible. But most of that was on roads (paved and unpaved) and just a little single track. My purpose on the bike was to photograph, so I was willing to compromise speed and agility of the bike to accomindate photography.
First off, I dislike greatly having anything on my back...don't like the higher center of gravity, the sweating wet back, the chance of falling and landing on my gear, nor the extra weight pushing my butt down on the seat. But my mtn bike had no suspension and rear suspension does make a rack and panniers difficult if not impossible.
I had two rear panniers for the gear and lunch, and the tripod fit nicely strapped on top of the rear rack with a couple bungee cords. With this set-up, just about any field 4x5 will do...just pad it well within the panniers.
If your bike has full suspension and can not take a rear rack/panniers, then think about picking up a used clunker hard-tailed mtn bike to return to places you see on your rides on the better bike.
I see a clash of objectives -- a desire for speed and a desire to get out there, fighting the slow-down type of photography that usually (but not always) seems to go along with view camera use. As long as photography is not the main reason you are out there on your bike, you might want to consider keeping with medium format. My first, biased, choice is a TLR -- especially a Rollei, then after that a folder. Light simple cameras with no extra lenses to haul around.