It all boils down to this...are we within our legal rights to shoot as we please. After that, it rests on personal etiquette, that which varies around the world.
This is entirely the point, and why the original complaint voiced by OP is absurd at best. If you truly don't give a crap about whether or not your subjects want to be photographed, and you are interested solely in the law, then surely you'd understand your subject is under no obligation to react positively to your actions and is within his/her legal rights to be upset, call you a pervert, a creep, an obnoxious asshole etc.
I don't think a Leica or dress code defines the pervert photog. What defines them is their portfolio. Even then, just because a photog has lots of kids pix, it is not a slam dunk pervert case. Sometimes women like to photo kids. They are not perverts, they just like kids.
It all stems from the likes of what we enjoy shooting. Helen Levitt had lots of kids. She just liked them and that was what was around her to shoot. I talked with one photog that found street boring. He liked landscapes. Another people hater like macro insects. OK, we are free to shoot what we like. That is the beauty of photography. (Unless your a paid photog, then you shoot as your boss paid.)