i am not familiar with the rules of the country the OP was in regarding street photos. but in the US we can still do it freely.
I am wondering if we should take a more aggressive approach? Should we start following our accusers down the street making our case argumentative style and getting upset at their accusations?
'Idiot...I am not a fudging pervert ...if i was, I'd be shooting from a van with a telephoto or a mirror spy attachment.'...type of thing.