My wife is an excellent baker. She can become involved in kitchen just as deeply as I can become involved in the darkroom. In fact, sometimes, while she's baking, I'll be in the darkroom. So, this proves that it isn't about understanding one another's hobby or passion. We're perfectly happy to give each other "space." However, she can ask me whether some recipe needs more cinnamon, for instance, and I can give her a coherent answer. But, if I ask her if a photo needs more contrast, I often get the feeling that she just can't see the difference.
I really think that most people believe that a photographer simply points the lens at something and presses the button while the camera does all the work.
I honestly believe that most people, even some who think they are photographers, don't have the ability to think visually, analyze what they see and take steps to make an image from what they see in front of them.
I think that is the source of the friction that occurs between photographers and non-photographers.