I shoot from 35 through 4x5 and have decent gear at the various levels and in my darkroom. Yes, you can get surprising results from well-crafted 35mm but the big difference is in the subtle tonal transitions between the formats. Each can easily have their place in your tool box.
I don't think it's accurate to look at format advantage as a linear improvement, marching along at a direct relationship to pure size. Many other elements introduce themselves as you increase format size. The equipment increases by cubic size, requiring more and more serious support systems, especially if controlling slr mechanisms and the mechanical commotion that goes with it. Optics are generally better corrected and controllable at wider ƒ stops in smaller formats, allowing for more reasonable shutter speeds for similar depth of field. Smaller film is more easily kept flatter and in the proper plane for best edge-to-edge performance.
A good, sharp medium or large format negative is a joy to work with and I find easier to print subtle tones (especially skin tones). They're sharper in the final product but when forced to deal with projects that only 35mm can handle, I'm usually surprised at how well the whole thing works out.
Each increment up in quality exposes a new issue to deal with. For instance, APO enlarging lenses help squeeze what a negative has to offer but their wider apertures require absolute, perfect alignment of enlargers (all planes) to take advantage of them. Raising the bar keeps raising the bar on your total routine and protocols. After 40 years of learning and upgrading equipment, I now have only myself to blame if things don't print sharp and precise! The reality was that I was at the core of most problems before becoming excuse-proof via gear acquisition!