I have never had a problem with anyone bothering me with a 35mm. I do have people who are also interested in photography sometimes stop and chat.
I also only get curious people coming up when I use 8x10 or larger. I think the size of the camera and the bellows makes a statement of hobbyist or artist.
With 4x5 I sometimes run into problems. People become suspicious that I am from the city government or IRS when photographing their structures. On the other hand I sometimes will employ a couple of traffic cones and a bright dayglo orange vest if I am working directly off a busy road. No one bothers me then. I believe they think I am probably doing some kind of surveying.