Diffraction increases as the pinhole gets smaller, which means that if you go smaller than the optimum size, you will get a softer image as well as longer exposure.

The optimum is the optimum because it is the point with greatest sharpness - too small and you're diffraction-limited, too large and you're limited by the tube of light coming through the hole.