I have not tried a telephoto pinhole, but I think your analysis is going in the right direction. You are effectively looking at a "crop" of what would normally be a much larger negative.

Pinhole Designer shows a 0.7 mm hole at 250 mm, using the constant of 1.9, but gets even smaller at a constant of 1.6 or similar values some people use. Going the other way, using the 1.9 constant and a 1.2 mm hole shows the optimum focal length to be about 960 mm which would be almost four times what you have -- probably ideal for an 8x10 or 11x14 (inch) monster camera! The optimum sized hole would help some, but I suspect it will still be disappointing.