I leave them on if they're not about to fall off, though when they're on the focusing ring of a lens (often) it's annoying. Call me an amateur -- I am! If they're long gone and the residue is there, I'll use naphtha to try and remove it. Interesting question as to whether collectors regard their presence as a sign of a well-cared-for piece of equipment.