I think have a low-level latex allergy: they're irritating to me if I wear them for more than a few minutes, so I can relate. I use nitrile or vinyl gloves instead, with no problems at all. Wearing gloves is cheap insurance that I'll be able to continue my photographic pursuits for a lifetime.
Have you considered using a barrier creme? There are two types you want the one that protects against water soluble chemicals. You can get it at most large drugstores. It's sort of like wearing invisible gloves.
A rock pile ceases to be a rock pile the moment a single man contemplates it, bearing within him the image of a cathedral.