My understanding is you set up to print in a closet. Closets are real dust traps. It should be vacuumed thoroughly; and of course, there is no clothing in there, right!?
And, be aware that the more you dust the neg, the more you can set up static charges that attract the dust. And I found, when using them, that the much-vaunted anti-static cloths never helped much.
I am not an electrician, so can't recommend how to do it; but you should check to make sure that the terminal into which you plug your enlarger cord is definitely grounded.
If there are just a few specks you see on the neg by oblique light, I knew guys who could pick the little buggers up with the tip of a spotting brush. I never could.