I've had this happen when I wasn't even taking pictures. Only just had the camera while sitting on a bench.
I was walking all day and stopped to rest. The camera shutter was locked and the lens cap was on.
There are just people who are THAT stupid.
Blow it off. Screw them. Move on. Life's too short.
In the future, you might find it helpful to talk to people.
Engage your people when practical and possible. Just talk to people about the weather or sports or what-have-you when you're not shooting.
Lone guys who sit on park benches are creepy. Guys who talk to people, who people see you talking to others, are social.
If you don't want to do that, bring a book and read it. Actually read it where people can see you reading it.
Bill Cunningham made (makes) his living shooting photos of strangers on the streets of New York.
Some people would definitely perceive him as a creepy, old man shooting pictures of strangers but I don't know very many incidents where he had problems. I say it's because he engages his subjects and talks to people. People actually come up to him and ask to have their pictures taken.
Bill rides up and down the streets on his broken down, old bike wearing shabby clothes. He just hops off the bike and starts shooting people's pictures.
But, instead of being accused of being a creeper, people invite him to their society parties and give him journalism awards.
Watch the trailer. The guy is amazing!
Second piece of advice: Be recognized as a photographer.
Carry business cards and hand them out. (Where practical and safe to do so.)
Go to the same places. Become a "regular." Let people know who you are and what you do.
I go to a place on the beach called "Sunset Point."
I've shot hundreds of pictures there and have made lots of prints.
Most of the people who also frequent the place know me as "that guy who still uses film."
I walk out on the beach and people wave. I know people by name.
Nobody there, not even tourists who have never been there before would question me because I look like I belong there.
I can shoot pictures of kids and girls in bathing suits all day so long as I don't make a nuisance of myself.
The last time I was out there, a mom, a dad and their two kids came out on the beach. Right about sunset.
Dad had a Canon digicam.
I was talking to this other woman whom I had never met. We were BS-ing about photography and what a great location this was for shooting pictures.
Dad and the kids walked by. He wanted to get some pics of his kids in the sunset. He couldn't get the kids to stand still long enough.
I came over and told the kids, "Hey! Go climb up on top of that sand dune so Dad can take your picture!"
The kids climbed up the dune. Dad started snapping away. I snapped a few. The other woman snapped a bunch.
I never met any of these people in my life and probably will never see them again but not a single person complained.
Matter of fact, we all smiled, chatted for a minute longer, shook hands and went home.
(Too bad those pictures were in the camera that got stolen. )
It's all about the way you act in public.