Eventually it will be the differences in the distance from the pinhole to the film at the center vs the corners that will limit you. If it is twice the distance to the corners than to the center, that will be two stops (4x) less light hitting the corners. I have made 8x10 pinhole cameras using 250sht boxes of photopaper. Very wide angle, but the corners can go unexposed relative to the center. Size of the hole will make no difference -- more light will hit the corners, but totally overexpose the center. So you need to know how much dropoff of light in the corners you are willing to put up with. A "focal length" of 3 inches will give you about 1 stop of drop off in the extreme corners of a 4x5, if my math is anything close to right (no promises!LOL!).
I'll shut up and let someone who knows what they are talking about post.