I have no problem photographing with others, but I rarely invite them to look through my viewfinder or ground glass. If they ask, I'll let them look through the ground glass after I'm done, but not before.
I usually like the conversations that happen.
But it sounds like from the answers here, that you guys shouldn't carry guns. Photo Road Rage, look out. Grin.