I think there's a common perception that if you consider something that other people use and like to have a shortcoming, you must be (gasp) a SNOB. And admittedly, there are some people who live right up to this stereotype, especially on the inet, and they kind of poison the well for the rest of us.

In my experience, I don't usually cause offence if I frame it as "I grew up shooting with fixed lenses, so even though I get what's so convenient about a zoom, I'm used to doing without it in practice."