You probably should not ditch everything for the Olympus XA. But feel free to pick one up and use it. Sometimes when you find a small unassuming lightweight 35mm camera that does all you need, you think that it's silly to want for more.
I'd rather hear you ditched everything for an Olympus OM-1 and Zuiko 35mm f/2.8 because the skill you pick up can be more transferrable.
Same film size goes in. But you can see more of what you are aiming at through an SLR than a small viewfinder can offer. And it isn't THAT much bigger or heavier.