Generally I've found people are just interested or excited, especially in the medium format cameras. I was shooting a Rolleiflex in a museum in Berlin last winter and a French tourist and his daughter walked by. He had some massive, expensive pro digital body around his neck, and he looked at me and then said to his daughter excitedly something that sounded like "that's a real camera!"
The only time I can think of where someone was giving me a hard time was when I was using a 4x5 view camera in the woods, within distance of a nearby path. Someone called out "are you shooting fairies with that?!"