Film wasn't that good back in the seventies, you could really tell the difference in quality. Having worked in the photo finishing area at that time though, the main reason that people didn't like them wasn't a quality issue, but it simply took too long for them to finish a roll....remembering that a lot of folks thought that 12 exposures were too many! Also, 72 prints made a big hole in the weekly budget.
Still, there is a lot to like about the cameras, especially those from Olympus as you have found. I have a Pen D,the EE3 like you and the lovely Pen F with a few lenses. The pen F has, of course been re-birthed as a Digital, so popular was it's design.