The function of the "condensers" (actually collimating lenses) is to make the light "bundle" more coherent - to take a lot of random light rays and get them going in a parallel bundle of the right size and in the right direction. The reflector in a flashlight serves the same purpose.
The effect is to provide an acceptably even illumination over the "field", in this case the "negative stage" (although it COULD be a positive). The most efficient condensers will be the ones designed for the format: 35mm for 35mm, etc.
Using a condenser set for 35mm with a 4" x 5" negative and lens would, most probably, result in a *lot* of "light fall-off" toward the outer edges of the frame.
Going the other way, 4" x 5" condensers for 35mm, would be less noticable; the exposure would be less due to the expanded physical dimensions of the "light bundle" - and the "evenness of illumination" would be ( or should be) somewhat less than with the lenses designed specificaly for the 35mm format.
Oversimplification, I know - but I hope this conveys the idea.