My red safelight is within three feet of both the dev tray and the enlarger base board I have no issues thus far with any fogging. Of course, a sheet of paper is in this range for no more than five minutes at a time, seldom for that long.
But, yeah. Once I added the roller cart into the mix thing start and end much more quickly. Darkroom work is much less a perceived inconvenience than it once was.
If I could change one thing in my present situation, it would be the location and arrangement of the trays. It's functional right now but there has to be a better way. I toyed with the plywood-ish idea, but I haven't come up with anything suitable just yet. A three or four foot by 15 inch card table with adjustable legs would be perfect.