Taking a photo of someone when they're in a private moment is an invasion of privacy, even if they're in a public place. Intruding into their private moment so that they become aware that you're taking a photo is an even greater invasion of privacy. We may have the legal right to do this, but that doesn't stop it being an invasion of privacy.
Personally, I believe that it would be a tragic loss if future generations are unable to see how people lived in 2013 because no-one made photos of strangers. This is one reason that I'm relaxed about people (or security systems) taking photos of me when I'm in public and I'm not aware that they're doing it.
But once they've broken into my private moment, they've engaged with me, and I think different rules apply. Once I'm aware of them, I feel that I should have a choice about whether to take part in their photography or not. My choice is almost always no, but others may choose differently.
I don't often shoot on the street, but when I do I treat others as I'd like to be treated myself.