Yes. Usually it is enough to listen patiently and let them know you will try not to include them in your pictures if they don't want. But I did have a mother call the police on me once because she insisted I was not allowed to shoot photos at a soccer game, though lots of other mothers and fathers were. Since I was using my Pentax 645Nii and a tripod I think she felt justified in causing me trouble since I couldn't have been a parent or grandparent. If I would have been using a little point and shoot, or my cellphone, she probably would not have paid any attention. I think her major bug was that I wouldn't delete my pictures nor would I show her my images. Since I was shooting film that wasn't possible but I don't think she understood the concept of film. She became upset enough that she stood right in front of my camera. I was using my 120 macro at the time and I got a great, tight portrait of her face, and that certainly didn't make her happy. She insisted she had not given me permission to photograph her but I pointed out that since she was preventing me from taking pictures of my grandson in the soccer game I may as well get shots of her instead.
The police arrived, explained to the nice lady that I had every right to take pictures of the soccer game, and also explained that since I was using film I would not be able to show her my pictures (I assume so she could personally "edit" them). Later on I did give her a nice 8x10 shot of her son kicking the soccer ball, which she did accept, but she still gives me dirty looks and this happened two years ago. On the day this all happened I just thought she was having a "bad hair" day, but I now think this is her normal attitude.
Life goes on.