Think about a teeter totter. With a 35mm camera with a longish lens you've got all the weight on one end. The leverage pulls all in one way. With say a 4x5 camera odds are the whole thing is centred over the tripod head. All nicely balanced.
Even with something like a MF camera odds are more of the weight is behind the lens. Heavier MF camera helping balance things out.