I think you're right. I recently took the plunge with 4x5, and will, as soon as my wallet can allow it, go for even bigger formats. The added sharpness is considerable.
But, why do you have to print your images so large? I always thought that 35mm and MF was best suited for 5x7 and maybe 8x10 prints. Like you pointed out, a lot of the inherent sharpness is lost due to the vast magnification. Already a 5x7 is quite a large enlargement for 35mm.
To me outstanding quality can be produced from 35mm and especially MF. But the print size must be kept down, even with ultrafine film like TechPan and APX25.
My 2 cents,
Saint Paul, MN