Some advice: don't begin with cheap junk because it may turn you off quickly; similar to drinking cheap wine and smoking cheap cigars.
Around $100 will get you a Yashica Electro 35 GSN, maybe a Canon GIII 1.7, or a Konica Auto S2. All three have decent lenses and rangefinders.
Around $300 will move you into the bottom of the interchangeable RF cameras, like the Bessa R and the Canon P or Canon 7 with a decent normal lens (not a Leica optic, though). Probably the best bang for the buck is the Bessa R or R2 with a Canon 50mm lens. I would advise avoiding Soviet bodies, as they are notoriously unreliable.
Starting at around $500 you can find some Leicas, like the IIIc or IIIf with an inexpensive Leica 50mm Elmar. The best thing about Leicas is that if you find out they are not your cup of tea, you can easily sell them and not get hurt financially.
But be careful, Leicas can be addictive!
Best of luck.