So my question is, how do you stick a camera in someones face without them picking a fight with you?
Know your equipment so well that you take less time to make the photo than it takes for them to get a complete sentence out of their mouth.
After that, just walk away. Or stay put. If they harass you, call for help. If you weren't meant to be making photos there but decided to anyway, walk out of that area (that is, if you start getting questioned about it.)

Don't think just do. From my experience, once you think, you've lost.

Have the police on speed dial or something, just in case someone does pick a fight with you.

Also, try not to have too many things hanging off of you, just in case someone really tries to chase you home.

Always be aware of your surroundings. An obvious one would be something like "don't stand in the corner and make photos if there's a chance your subject will harass you, because then you have no chance of escape."

I always feel like Im a prisoner on the run HCB