I do a lot of street shooting, so that is not uncommon for me. I have even been chased by a gang that was scavenging scrap metal from buildings to be demolished. I discovered that developing strategies aimed at defusing confrontational situations is a key part when doing street shooting, as it contributes a lot to what one can or can't do. Many times I simply refrain from taking the photo just because if things go badly wrong, I won't have a viable exit strategy.
Most of the time I go unnoticed, as I don't do Gilden-style street shooting, but I have friends who do, and they get themselves pretty soon into tense situations.