Another way to think of it is as the amount of light (energy) per unit area. The hole of the aperture doesn't limit the total amount of energy coming thru the lens, only the amount proportional to the full aperture setting. Any given object in your field of view reflects a given amount of light energy (a given number of photons). But the more objects that the lens sees (its field of view) the more total energy (total number of photons) gets reflected to the lens. More photons spread over more film area comes out equal each time. This is why exposure meters don't need a setting for the film format.
I proved this to myself the hard way after spending a bit too much to buy an OM to Pen lens adapter. In an ongoing attempt to photograph the moon, I was hoping to take advantage of the Pen's 1.45X "magnification factor" without the degradation caused by a teleconverter. Of course, for a given lens the image on the film was the same size whether taken with the OM or Pen --> Same moon, same number of photons, same exposure on the same area of film.