I currently own and use a Rollei 35 (Tessar), a Retina I (010 w/Schneider Xenar), and a Voigtländer Vito (Skopar) - all small, pocketable fully manual cameras with 50/3.5 lenses. They each have their charms, I've been happy with the images they produce, and they are certainly easy to carry (the Voigtländer in particular is light and very well-designed for carrying in a pocket). They or similar cameras are a viable option if portability is the primary consideration. FWIW, I do not find the Rollei's ergonomics problematic, but some do.

That said, I do not think they are a true substitute for an LTM setup. Lacking rangefinders and without interchangeable lenses, they are not as flexible. You might look for something in between like a folding RF (the Retinas offered a 50/2 Xenon that has a good reputation). It's simply a question of priorities.