Contrast can change by a number of theoretic mechanisms:
1) psychologic impression of the image
2) More fog from a leaky head. A 150 second exposure can fog the paper more than a 5 second exposure if your enlarger head leaks a little light (they all leak some).
3) More fog from stray light bouncing off the paper and coming back on your highlights. 16x20" print (320 square inches of white paper) reflects more stray 'white light' into your darkroom than 4x5" (20 square inches).
4) Slight variations in the paper batch for a different size
5) Potential for less developer activity on 320 square inches of paper vs. 20 square inches
6) If you have to open the lens from its 'sweat spot' it will have more flare.
7) A bigger print requires one to use the far edges of the lens' image circle. The MTF is lower out there and contrast can be lower also.
On the positive side there could be less flare in the enlarger head because the bellows is collapsed more.