No offense taken. Get some rest, get up fresh and have some fun. There is not much more rewarding than a good LF contact print. I promise it will be worth the headaches.
It's the glass, not the light source. (Also, you can print and filter VC paper with a regular tungsten globe, you just have to rig the filters up.) Take the lens out of your enlarger, board and all, and you will have a larger source, but not, of course as large as a diffuser placed directly on the frame, you will still have nearly the same problem. You will see that the source has to be nearly as big as the frame before the imperfections disappear. Now that is fine for straight printing, however, eventually, when you want to dodge and burn you will want to get between the light source and the print, so you will have to go to some kind of harder source. You gotta have good clean glass. I suspect you live in a dry climate, since dust seems to be a problem. A humidifier in your work area will go a long way to holding down the dust level, as will an air purifier in addition. I have a big Honeywell that I have on a timer, runs for several hours at night, every night. Cheap, compared to wrecked prints. Long ago, in a galaxie far far away I fought the dust/scratch thing at first too. Had an old antique contact frame, weak springs, bad glass, dusty darkroom. I tried to work around it, but in the end, I found it just had to be done right.
PS I've seen dust in Ansels skies. Don't forget that spotting is part of the process. You just want to minimize how much. Rarely will you ever make a print that is perfectly free of any imperfection. It is a watermark of the process.