Let the vibrations ease off for a few seconds before you start the timer. Don't touch the enlarger or anything connected to it while exposing. There should not be a reason to touch anything during the exposure anyway.

Another source of loss of sharpness is negative pop. During long exposures the negative gets hot and pops in the opposite direction. If that happens, you need to let the set heat up past the popping point and then expose.