I use either a slr with a noisy motor drive, the most conspicuous lens I can carry and a monopod, or the little Zeiss ZM rangefinder with a 35mm lens and a 90mm in the pocket.
It all depends on what I'm after:
If I'm out taking deliberate photos of some event or to get a specific result, it's the slr.
If just casually walking around at lunchbreak to see what others are doing, the rangefinder.
Mostly I take photos of either parks and folks relaxing in them, old buildings and their relationships to each other and the environment around them, or pure people street scenes.
So far only had one instance of anyone putting up a hand and saying: "no photos, please". Which was promptly and courteously acknowledged, accepted and followed.
I've had a railway station security officer ask me rudely "what I thought I was doing". That ended up in a formal written complaint to the railway authority and a written apology sent back to me as the result. Haven't had a problem in railway stations since.
I get the odd "filthy look" from folks who obviously can't think. Those I just ignore and that's the end of it.
Sometimes I get weird looks in the beach when carrying a camera. I only have the camera when my kids or my wife are around, so I just take a photo of my own family and that ceases the "another weirdo" look straight away.
Oh, and there is a local coffee shop owner who always likes to pick my F2 or whatever other film camera I'm taking and play with it: ex photographer, he loves anything non-digital.