I have seen a safelight cause this first hand, not enough to fog paper but enough to prevent higher contrasts being achieved, Paterson released a VC safelight to prevent this.

Try a better safelight with a dedicated VC fliter and make sure the distance is more than the recommended minimum, use a dimmer switch if needed.