Conservation of light, i.e., what area is the light spread over, is certainly pertinent. The other physical factor is that as the lens is refocused the distance of the entrance pupil from the film changes. For the small print with the small enlargement factor, the light is spread out over a small area, but the lens is focused farther from the negative and so the entrance pupil is farther from the film and therefore captures a smaller fraction of the light that is emitted by the film. This effect is usually termed "effective aperture". This is still conservation of light -- as the entrance pupil is farther from the film, some of the light rays no longer enter the pupil.
The effective aperture change when the lens is far from the film is also why one must correct exposure when taking a closeup photo. This is most commonly encountered by large format photographers ("bellows extension"). (Cameras with through-the-lens metering do the correction automatically.) In that application, the equation is highly accurate.
For the enlarging application, the equation is implicitly making some assumptions about how the film emits light, which of course depends on the illumination system of the enlarger. So the equation might break down in extreme cases, such as point source enlargers or very wide apertures. It should be most accurate for diffusion enlargers with the lens at reasonable apertures for making a print.
For someone who wants a calculator/computer implementation of this, there is a paper dial "computer" in the Kodak Darkroom Dataguide that gives the change in exposure time going from one magnification to another. You can probably find this inexpensively on eBay or internet used book sites.