You should indeed not use the flange to film plane distance.
What you should use however is the pupil diameter, i.e. how big the physical hole size appears when magnified by the glass.