I've owned the Voigtlander Bessa III version of the GF670, and can vouch for its compactness vs a Hassy V series. Plus, you can shoot 6x7 and 6x6 with it.
I ultimately decided to part ways with the Bessa though because of some issues that cropped up when using it in very cold weather. The rangefinder mechanism essentially broke in the middle of a shoot, and it took me months to get "repaired" (long story... bottom line, I never got "my" camera back from the dealer, but rather a demo model). I've been using a Mamiya 7ii ever since, and from the perspective of switching out a 500 system for a rangefinder, I think you will want to consider the usefulness of being able to swap out lenses on the Mamiya, vs the fixed lens on the Fuji. You do get a little bit of an extra wide aperture on the Fuji (f/3.5 75mm vs f/4.0 80mm), and it does fold down pretty flat, making it a truly portable, carry in your jacket pocket kind of camera. But I decided to go with the Mamiya, and after switching, wouldn't go back. It is a much more solid feeling camera, the rangefinder mechanism is a lot more protected from the elements, and there are 5 very good quality lenses to choose from, as opposed to the fixed 75mm on the Fuji.