Paint the walls an off-white. You'll be able to get a much better distribution of the safelight with no effect on the paper as the safelight will be very diffuse the light is actually very weak but more evenly distributed. I've worked in a number of different darkrooms painted: grey, red, and white.
White is by far the easiest to work in (for me) because of the more even safelight level. Paint the area in back of the enlarger flat black, or hang a piece of black cloth on the wall to absorb light that might be reflected from the area around the easel or escaping between the carrier and the lamphouse.