Yes, dust is a real plague in the darkroom. The only method to get rid of it is to clean all the tables, boards, the floor and the enlarger with a damp cloth every time one uses the darkroom. uh! But after this the joy for darkroom work almost is I donīt do this myself ;-)
Maybe you could wet the table around the enlarger before removing the plastic, then the dust will stick to the surface of the table and can easily be removed.
I always do this before spooling film in the tanks to avoid dust on the negatives and it works fine.
Then clean the enlarger with a damp cloth. This should solve the problem.
Greetz, Benjamin