I too wear cotton gloves, but when needed, I wipe the area being spotted with a cotton swab, which is on the other end of the brush. Let me explain.
Take some ordinary cotton watting and pull it apart so that it is spread out much like an air filter would look. Then wind it onto the other end of your brush, sort of like cotton candy. It'll look like an over sized Q-tip. When done, give it a lick and smooth it out with your fingers. This will keep the cotton gloves clean. The quick reversal of the brush in the hand to wipe the spot will become second nature.
I also use a lamp with a built in magnifier, you know one of those with a ring shaped flouresent tube. They are of great help providing bright, even lighting, and the lupe sure takes the strain of the old eyeballs. If you buy one of these babies, don't go with a cheapie. The lens will difract and the adjustable arm will not stay in the proper position. I know, I made this mistake once already.