If you must touch it up, go to a shoe repair shop or leather worker first. As you say, it sounds like rubbing on a shirt or such. If you use the wrong coloring, it too will rub off... onto your shirt. I know that using shoe polish on camera leather or on cases is discouraged for this reason.