To a point the smaller hole produces a sharper image, but eventually the sharpness becomes diffraction limited. Probably equally significant is the quality of the pinhole itself. A perfectly round, knife-edged hole is the best. Pushing a needle through something creates all sorts of turned and ragged edges. One classic method just uses the needle to raise a small pimple, then that is sanded off very carefully to open a hole. With that method, the needle diameter isn't particularly important.
There are some calculators and resource pages on the web you might find useful.
Sounds as though you need longer exposures. With some films, the reciprocity effects become extremely severe, needing 3 or 4 times (or more) exposure beyond what the exposure computed from the f-stop might predict.
It's fun stuff.