I'm not using the most high-end material for my pinhole, that's for sure - aluminum from a soda can with a hole either drilled while clamping the foil between two thicker aluminum pieces (for holes >1mm) or the old pin-and-sandpaper method (<1mm).

I don't know how much a fine edge will affect a hole that size though.

@Andrew, you brought up light color that might add further to the fuzziness (I didn't know about that). I have tested on night street scenes, lit with mercury lamps, thus heavily biased toward the red end of the spectrum.

So, as far as I understand the optimal pinhole radius is the point at which the diffraction and the circle of confusion balance each other. If the distance from pinhole to film increases, light rays diverge further and the diffraction effect becomes more apparent, and you need to open up the pinhole to compensate that. But if you keep the same frame size, the COC becomes larger on the film.
So, apparently the "optimal" pinhole size for a 10 FOV is less "optimal" than the one for a 50 FOV - correct?

Anyway, here are a couple of shots I took using a digital SLR - which means the FOV is even narrower, comparable to a 320mm for a film camera. As you can see, the results are pretty unusable. Pinhole sizes are 1.2mm and 1.8mm (the one I would like to use for exposure purposes).
telephoto_test-250mm-1.2mm_hole.jpg telephoto_test-250mm-1.8mm_hole.jpg