You have stumbled upon the very reason for the existence of the Banquet Camera. These were used to photograph large (very large) groups of people where photographers quickly stumbled upon the "too little real-estate" to resolve detail. If your subject's face occupies a sufficiently small area of the film, you may have an insufficient number of silver crystals to resolve the details. For groups of say a hundred people, a 35mm film frame may have one or two grains per face.

Asking your film to offer the fine grain to resolve something so small is also going to stretch your technique and lenses. If you have those, try some Technical Pan, or some Pan F.