I like to think of it as needing a certain amount of photons striking the emulsion to give a desired exposure (reaction in the chemistry of the emulsion). If you halve the shutter speed you double the aperture size (whatever combination of settings being manipulated, including the rated ISO of the film, are adjusted in relation to each other to achieve the "balance" desired).

The numbers themselves come from the math, and that is where I lose it

My boss in my old position at the university I work was also a professor in the Math department. We joked that no one could let him know how bad I am at math.