I good technique I've found to minimize the distraction of questions, or even of confrontation sometimes (though some of my best shots have been taken in the moment of confrontation.) Is to wear a set of the big over ear headphones. Sometimes it can be nice to listen to music while shooting and other times I want to be in tune with the sounds of the street, but even then often I'll leave them on and just pretend I can't hear anyone who is bothering me. Most of the time people will just leave you alone because they assume you can't hear them anyways.