I have never had issues with the meter on any Olympus camera, and I've got a pretty wide sample to choose from. Like all Zuiko lenses, the XA produces very contrasty images. Someone not familiar with this would likely think the shots were underexposed. My XA2 is even more contrasty than my XA, and there is an active thread inquiring whether others have observed this.
I would not recommend Velvia as this is a very high contrast film to begin with. If you want to stick with Fuji, I would try Provia or Sensia. Ektachrome and its brethern work very well.
I'm considering putting a roll of Velvia through the XA for 'life' shots.
Is the meter in the camera accurate enough for slide film?
Yes, definitely. The one I had went from 1982 to 2006 when I changed to digital. In the 1980s, it was Kodachrome going through it, then Ektachrome and latterly Fuji Astia.
In common with widespread practice with contrasty film (i.e. Velvia), I would consider rating Velvia 50 at EI40 (40iso) and keep the needle out of the over- under-exposure areas. Shoot only in diffuse / hazy / overcast light to improve your chances of a good image; Velvia 50 is not designed for bright sunlight.
When I saw the post title I said "this guy's nuts." Then I read your responses. Wow. I'm on the bus home and have an XA in my coat pocket that's intermittently screaming "it's too damn cold" and "Velvia 100F!"
"There is a time and place for all things, the difficulty is to use them only in their proper time and places." -- Robert Henri