1. Diameter of pinhole. The smaller the diameter, the sharper the image.
2. The material the pinhole is made out of. Shiny thick brass plate or sheet metal yields fuzzy image. If any backlighting conditions exist, it causes reflections inside the pinhole "tube" and causing major swirls and refractions or aberrations on the negative. Thin, dark, opaque material makes sharper image.
3. Dimmer conditions where lighting is more even (such as interior shots with ambient lighting) make for long exposures but very sharp images.
4. Bright lights tend to scatter light inside of the cheeper or home built pinhole cameras and while faster to expose, result in more fuzzy or rather less contrast which makes it look less sharp.
5. Camera shake. Or wind shake, like photographing trees outside in the wind with a 3 minute exposure. You will not get ANY sharp image that way.
Likely I am missing some more.