I norrmally change lenses.
Lets think about this. What do the condensors do? They focus the light onto the negative. With the enlarger setup for 6x6 the light is focused to 6x6. Most of that should just be hitting the negative carrier with 35mm.
The only thing I can suggest is put the lens cap on the lens and turn the enlarger on. Do you still see the image? If you do then you've got some sort of leak.
I don't have a 40mm to test here but I just tried 35mm with the enlarger set for 6x6. Nothing I could see.