Depending on how flexible you are with the interchangeable lens clause, I would definitely consider a TLR of some form. Minolta Autocords and various Yashica Mat's are worth looking at in the budget department. Rolleiflexes are fantastic if you can stretch your budget a little more. I've also heard good things about Ikoflex's, although they may be more difficult to find in operating condition. I think TLRs are the best introduction to medium format, and they are really a different experience than shooting 35mm. Everyone should own at least one!
Mamiya C220/C330's is a TLR system with interchangeable lenses, but I find them a little clumsy and would go with a SLR system instead. Medium format SLRs tend not to be small, per se, but Bronica SQ series and Hasselblad V series are both reasonably sized. I personally prefer 6x6 ratio over 6x4.5, and the SLR sizes are not hugely different in this case. You can pick up Mamiya m645's with multiple lenses great prices though. 6x6 will feel more 'different' than 6x4.5, not just because of the increased resolution but because of the unique compositional elements.
Medium format rangefinders with interchangeable lenses such as the Mamiya 6 or 7 would fit the bill, but they are massively more expensive (even than high end SLRs). I've never used a MF rangefinder, so I can't speak to if it is worth it, but they certainly have their fans.