I haven't moved to colour yet but even with black and white I have a big 100 pack of disposable gloves and use a new pair each time.
I'm not convinced that an occasional dabble is going to wreak havoc on me but there are certainly things in photo chems that I would rather not soak my skin in.
Disposable gloves are a cheap precaution, I would use them.
Think of them like condoms....you might never have a problem but if you do you'll wish like hell you took precautions!

Besides, your hands don't stink of chemicals for hours afterwards...