The key point here is not what color the walls are, it's what color the walls reflect.
If all of the possible sources of light in a darkroom are safe (by whatever definition of "safe" is required for the process being performed), then by definition the walls—regardless of their color—cannot reflect an unsafe color. In this case any color will do. But white is preferred to spread the "safe" light around more efficiently.
The problems arise when one has white walls and the enlarger leaks imaging light. Or the doorframe leaks white light. Or the safelights aren't really totally safe. Then white walls only make the problem worse. In those cases black would be the ideal color, because you'd want to limit the spread of the "unsafe" light as much as possible.
Regarding the OP's question about UV causing optical brighteners to emit light from the walls, I've never heard of this being a problem. But the guaranteed solution would, of course, be to simply add UV light to the set of "unsafe" light colors and eliminate it from the darkroom the same as regular white light. And amber/red, not being UV, should have no practical effect.