I have a set of kodak filters that clip on also, mainly used by me for its strong #5.
But I was really referring to using using the red swing in filter for contrast filters. If the filters are bent or damaged in the process of exchanging these frequently they might affect the optical path if they are not in motion thoughout the exposure.
If they are bend or damaged, I'd through them out. Under-the-lens filters must be clean, dent and scratch free, or, as you say, they may affect image quality.