More power to the so called lesser family member. Use what you like. I always take advantage when a model I like is denigrated by buying up an extra copy on the cheap! Sometimes the prestige models aren't worth the premium over the less glitzy ones.
It's really the glass that makes the pictures anyway, not the body, and Olympus glass has nothing to apologize for. I mostly settled on the '60s-'80s Nikon system these days because of the wider availability of used gear, but my experience with Olympus lenses was spectacular. Most of my lenses are worth more than the bodies to which they are mounted.