The condenser focuses the light to the center of the enlarging lens. So for a 6x6 condensor you are using either a 75mm or 80mm lens. When you switch lenses for a smaller format the condenser will still focus the light for the longer lens. This will be exaggerated with a 40mm (wide angle lens) as the focus point will be ~40mm beyond the lens. This is going to show up as a weird light source outside the image area. It will not be as exaggerated with a 50mm lens.
Did you notice you had to rack the lens a whole lot closer to the negative stage to focus the negative?