I did a lot of street photography in NYC a long time ago. The place is so busy... even pictures I would stop and compose for (often it was people lined up to get on a bus for some reason) they wouldn't really notice me. Sometimes, I had the camera around my neck, and would keep it there, having it set for the light, and focus at infinity, and just release the shutter. I wouldn't necessarily know what I got, but I have a few great pictures made that way. Also, in winter, when the light is low, walk the streets so that the sun is behind you, but in the eyes of people walking toward you. They'll have a hard time seeing that you even have a camera.
I think the best way to avoid pissing people off, is to work as a kind of "fly on the wall" so that you aren't really noticeable. And if someone does see you... question you, tell them honestly what you are doing. It can be done if far less intrusive ways than Bruce Gilden.