People react differently to different cameras. TLRs seem to get ignored or smiles. SLRs are more noticed, sometimes more threatening, sometimes confused for video cameras these days. The important thing, as mentioned again and gain, is your comfort, though. If the Yashica is doing it for you, and all you want is a slightly better lens and a more solid mechanism in the same form, Rollei it is.
By the way, look hard at the f/3.5 Rollei Planars or Xenotars. That half stop loses you little in practical use. I have a 2.8C, and I can't remember the last time I used f/2.8. The 3.5s are cheaper, the lens is as good. And to save more money, go for the Xenotar- 'the same' quality (some will debate) but less status and hence less money. An E series is also a bit cheaper than an F series with little or no lose.
For either Hassy or Rollei, condition is as important as anything else. Look into maintenance needs and what an overhaul can cost and include that number in your figuring.