One issue is that the smaller the format, the easier it is to carry more film, and the less costly the film. I think this is the major difference. With 8x10", 10 sheets would be a productive day for me. With 4x5" and a few Grafmatics, I feel like I've got my sheet-film motor drive on.
Another is the larger the format the longer the lenses, and the shorter the DOF, so a little more attention to focus may be required, but not that much more.