Essentially, what you want to avoid, Jeffrey, is a situation where white light from the enlarger can bounce around, potentially reflecting off some other surface back onto the print you're exposing. As such, I'd keep the side baffles on the adjustable section. Like Konical, I like the enlarger area to be flat black, but the rest of the room a neutral, non-glossy light color.