Plated gold contacts are surprisingly tough. They're usually plated onto a barrier metal, typically nickel, to improve adhesion to the underlying (usually) copper. Back in the old days it was not uncommon to take an eraser fairly aggressively to the gold-plated backplane fingers on printed circuit boards to fix intermittent contact issues - this was the fix of first resort in my first EE job out of university.
That being said, I used the eraser very gently and with minimal pressure. A cotton cloth dampened with isopropanol would probably have been even better, come to think of it.