Don't know if that is true. (US) You can say f off or get in a fight. But the photographer right to shoot is higher than your right to refuse. (at least that is my understanding. correct me if i am wrong.)
Unless you are famous, you have some rights to privacy. People just can't just "stick a camera in your face". You can shoot the street, but "zooming in' on an individual is risky.