So if the lens-flange face wears down over time, then eventually the elements will come microscopically closer to the film flange. But that is offset by the fact that the rangefinder flange also comes closer to the film at the same rate, so it all evens out in the end. Except that at 'infinity' on the lens may be past infinity, but other things being equal the rangefinder will still be accurate.
I'd be more concerned about the rangefinder cam and/or the flange on the lens getting worn out. Mount and unmount all you want, just don't change focus too often!