Maybe you could look into the Soviet Era range finders. I have a Fed-3 and it takes interchangeable LTM lenses. They generally come with a sharp Industar 50/2.8 lens. They can be had on ebay for about $50. They are in between in size of the Canonet QL you tried and a Leica M. If you want to move up to an Leica M camera in the future you could use the lenses you've accumulated.
For a faster lens you could pick up the Jupiter-3 50/1.5 for around $50 US.