I'd have no problem with using one per se, but that's assuming I could afford one. Being inconspicuous is one thing, but there's always the problem of people taking offence at having their photo taken and attacking you (and damaging the camera in the process), or just anyone noticing your expensive camera and stealing it. That's why I went for a Bessa. Average Joe with enough brains to have to resort to stealing things for a living won't know the difference between it and a $4k Leica, but from my end if it gets stolen i've lost $500 and a roll of film max...

Seeing as the xpan can still shoot normal 3:2 shots, I wouldn't call going wide a hinderance. Even if you leave it in the wide setting all you do is burn through film faster, you can always crop later. Whether it adds anything to the shot I don't know, that will be very subject-dependant.