I mean, what if you had not acted fast enough, or made a move they didn't like, and they'd popped you? It hits the news: six armed Saudi hotheads from the embassy shoot an American citizen on American territory in the nation's capital- film at eleven. I would expect our State Dept. to think about these things...oh well.
Try walking round Marrakesh with a Leica during Ramadan. I was given a bollocking by some local Herbert whilst taking photos of a door (no-one remotely in danger of being in the shot). Not being the smallest of people or a shy, retiring type, I taught him some Anglo-Saxon phrases he probably hadn't come across before.
Later in the week, I had a couple of guys demanding money for being in a "general" shot of the street we were walking down. They got short shrift as well.
The world is full of miserable sods who will chance their arm in the hope that we will shell out cash easily. If you're just shooting a general scene in a public place, then tough it out. If you want to photograph someone specific and you want to get in close, then it's only polite to ask their permission and to accede to their wishes, especially if they say "no". Candid shots can be fun but they can also be very hit and miss.