The subject can determine the format. But your concept can determine it too.
There are photographers carrying very heave large format cameras and accessories up to mountains and long paths on the middle of nowhere just to take some portraits of local communities. They culd carry a 5x4 camera or a middle format one. But no... They want a ultra large for it.
If you really want you can shoot action with 5x4 large format camera handheld.