That's part of the gig sometimes. I can honestly say that I certainly received more negative feedback when I have attempted street shooting, in the past, with a big SLR or DSLR (and forget about MF obviously). I now exclusively use my M3 and an old Summicron. I rarely get noticed and, when I do, 99% of the times is a smile (maybe it's because I'm so handsome...just kidding).
Two weeks ago I was in NYC shooting some Kodachrome 25 and noticed this pretty young lady, with a beautiful flowery yellow dress and a feather in her hair. She looked like straight out of the late '60s. I was using a 50mm and get as close as possible, snapped a couple, then she turned around, smiled, I smiled back, and let me snap two more. Another guy who noticed that, tried to get a shot with a Canon DSLR fitted with one of those cannons...he got the finger as soon as he lifted the camera to his eye. I don't think he got a shot off.