The "image" is formed by the light moving in straight lines. So the resolution is inversely proportional to the size of the hole. Same as with lenses; just about anything gives a decent enough image at f:90 while it could be wholly unacceptable at full opening. Now imagine a really, really dreadful lens. Stop down to f:360 or so, and the image will be reasonably sharp and DOF really great. Then discard those useless bits of glass. Now you have a pinhole! No refraction necessary at all, but shrpness will deteriorate both by stopping down further (from diffraction) and by opening up (from the dreadful focusing capabilities of air).

So to make a good pinhole, it should be small. The "circle of confusion" is directly proportional to the aperture of the pinhole, so smaller is better. But every hole has an edge, and edges give diffracton. So the closer together the edges are, the more diffraction will be visible - because the total amount of lihgt entering the hole gets smaller, and a larger proportion of it gets diffracted.

There are lots of recommendations for the optimum size of a pinhole, unfortunately they have all fallen through a hole in my memory...