I just see 4x5 and MF as two completely different worlds (with the Fuji MF system that has movements being a strange & "wish I owned that" hybrid).

I just don't think you can "replace" one for the other; the process, gear, and results being markedly different.

Not saying "don't do it", just keep in mind that if downsizing is the deciding factor, be prepared for a very big change in how you make your art (or enjoy your hobby or produce your client work). My 4x5 gear all fits in one case (that hasn't been opened in a few years to be honest) so this doesn't have to be an "either-or" deal unless money is the major deciding factor. And even if budget is driving your decision, you could likely find a used RB, 645, Rollei or even a Kiev and shoot a few rolls to get a feel for the process & the results.

I'd vote for trying to have both systems available to you, at least for a while.