It's also helpful to know that all optical design has tradeoffs. Sure, you can get a sharp 1.4, but to get that usually there is corner falloff. They have to make choices. It's just the way optical physics works. There is no optimum design. price and size also impact.
Also, as the colour film (and TV) era dawned, many fast lenses of the 1970's to 1980's were actually designed to be less than sharp, and one way to do so was shallow DOF. Colour film was often too revealing of blemishes etc., so "soft" became an aesthetic.