I agree with Parker Smith about cleaning the negative before placing in the enlarger. But, it does not hurt to have everything as clean as possible so that there is little to do to the negative. I am fortunate to have a dedicated, purpose-built, darkroom. All seams in the drywall and trim are caulked and painted. Semi-gloss and gloss paint for all surfaces (this helps with odors, too), and a tile floor. I flush and wipe out the sinks after printing. Those dried spots of developer, fixer, etc. not only add "character" but they also add dust. The floor gets swept and moped periodically.
For the negatives: first, they are stored properly and kept clean from the time of processing. When mounting in the carrier, I use a camel hair brush, which gets almost all dust off and is good enough for proofs and early work prints. (I am going to try PEC pads however.) When I get to making "public" prints, I use low pressure air supplied from a compressor - not canned air. The compressor is outside the darkroom with just the air line coming through the wall.
Do not under-estimate the effect of how clean your enlarger is - or is not. One can scrupulously clean their negs prior to insertion in the enlarger, and then have all kinds of crud settle down from inside the enlarger when the negative stage is closed. How do you know? Just turn the lamp on. open up the negative stage without a carrier and look inside!