I've had the same problem in the past but wasn't smart enough to think of tape.
I used a paper clip bent in a "U" around the back of the finder and the front ends bent down to keep it from just sliding off.
Heat shrink tubing on the tips keep them from scratching.