Just a quick though, and you may have already dealt with this, but...

When I use my enlarger as my light source for contact printing, I leave the lens in so I can control the light via height and f/stop. But I also make sure to rack the focus (and now I can't remember if I'm racking it full in or out) so that the schmutz that might be on the condenser glass or on the filters won't be focused by the enlarger lens.

This probably isn't that important but if you were to leave a negative carrier with glass in the enlarger, you could potentially be focusing any of the dust on the carrier onto your final print. I stopped using glass carriers so my only concern is dust on the condenser lenses or VC filters.