The smallest real rangefinder is an Olympus XA, selectable aperture priority mode, up to 800 iso. Clam shell design means no lens cap. It's so small it fits in a pants pocket. Everything else is jacket pocketable at best. Terrific 6 element, 35mm, 2.8 lens. Uses modern batteries, not mercury. I got one on ebay for $35 with the flash.
You're right on the XA's size. I have had a couple of them as well. While the size is convenient, I had one just stop functioning on a vacation (rangefinder somehow broke/became uncoupled and the meter stopped working - I had it in my carry-on or with me the whole time and it wasn't dropped as far as I know). Also, the rangefinder patch is rather small (necessary for the size) and that can be difficult to use in low-light/indoor situations.
So far the Oly 35RC hasn't broken on me at random, and it has a decently visible rangefinder in most lighting conditions, so I tend to rank it higher.
My experience with the XA may not be typical re: reliability, but it has soured me on the XA relative to other rangefinders. YMMV, of course.
Do yourself a favor and go pick up an Olympus just to see how small and full featured they are. They are plenty small, and sacrifice no SLR features to be that small. IMHO, there is no reason to sacrifice the advantages of an SLR just because you want a camera that is not too big. What you gain in size (or lack thereof) with a rangefinder, you lose in other ways. Do not get me wrong. I love using rangefinders in the right applications. However, if SLRs are what you need to shoot what you need to shoot, then rangefinders simply will not do. Seeing what the lens sees is not a feature to be easily brushed aside in favor of something more trivial, such as a minor difference in size.
"Truth and love are my law and worship. Form and conscience are my manifestation and guide. Nature and peace are my shelter and companions. Order is my attitude. Beauty and perfection are my attack."
From the original post, I can not tell if you are looking for a SLR or simply any camera that is smaller and manual. The Olympus OM series are an excellent platform and you can do even a might betterf with their F/FT/FTV series. Now that is 1 small SLR. If other than an SLR the Leica CL or Minolta labeled version is my choice as it is real Leica quality on a budget. For something more compact something like a Zeiss Contina II with Nova shutter is difficult to beat. The front folds up similar to the Olympus XA, has a decoupled rangefinder that is easy to work with. Smaller? Look for a Minox 35mm but get a late s/n ML if looking at the ML. I prefer the MDC or late ML over the other models as these can be used as manual cameras.