If you're looking for complete manual control, The XA is AE automatic with a +1 stop compensation. From the picture the Contax seems to be the same and both are battery dependent.
For not much more in the size dept a Leica CL or Minolta CLE would do manual control, and interchangeable lenses. but honestly I think I'd rather plunk down the 400 for a Contax T and just not worry about it. My M2 takes care of the rest.
When I do not want to carry my Leica or my SLRs I grab my Contax T2 or lately my Konica Hexar AF. The latter not much smaller but lighter and simpler with only one fixed lens. But oh what a lens.
The smallest is the previously mentioned Rollei 35 but there are cameras that are only a bit larger, such as the very compact Ricoh FF-1.
Here is another vote for a Olympus 35RC. Auto exposure, you set the shutter speed, meter selects the aperture. Also full manual control with all shutter speeds and apertures available if battery dies or meter goes belly up.
While in comparison with an Olympus XA it isn't the smallest, but as a complete camera with everything the photographer could want the Olympus 35RC beats all others.
The 35RC is pretty much the same as all the other compact shutter-priority rangefinders of that era. You can't really go wrong with many of them regardless of which manufacturer you choose.