Smallest camera I was using when asked was my Minox LX. Admittedly it was mounted on my Majestic tripod at the time. I simply answered no and turned my back and got down to business but the gent would not take the hint and continued to talk. I finally pulled out my Polaroid 180 and told him it was a Hassey without opening it, put it back in the back before he could read the name and he seemed satisfied. He then told his wife he knew I was a professional and I turned and simply said, no I'm just rich and my other camera, a Leica I left in the Rolls at home.
As for medium format, I'm a bit more forgiving with the Bronica, especially when using the waist level finder as their general shape is similar.
I tend to get a bit more annoyed with guys and gals asking if the camera is a Leica, especially when using the Bronica or Rolleiflex. When using the CL, I sometimes answer that it really is an Olympus 1/2 frame that I painted and put the Leica name on. I love it when the response is something like, "I thought so".