I have read about this problem before. There is a sliding contact that moves with the f stop setting that gets intermittent. This is one of those rare cases where lack of use is bad. Normal use of the camera keeps the contacts clean. I have never had any problem with my EM, and it gets lots of use. I suggest you "dry" shoot some pictures without film, as if you were shooting a few rolls of film. See if the intermittent problem goes away. It is a neat little camera, I hope it works for you.
Best regards,