I've had cameras exhibit this as well, alcohol or naphtha usually cleans it up. I've also had cameras where the rubber develops a white coating I guess from drying out too. That's usually much more manageable.
When I used to swim a lot in school the recommended way to prevent the rubber caps from sticking was to use a little bit of talcum powder (baby powder) and rub it in. Maybe that may work.