Olympus OM-1 and 400mm telephoto

This week end I am planning on shooting pictures at my neighborhood garage sale. For some of the shots I want to use my 400mm Soligor. But I find that the metering on my OM1 makes no sense. It reads as though I have lost 4 stops of light even at f/ 5.6 in bright sun. I had this happen before on my OM-10 on automatic and had to switch to manual. I thought it was just that camera. But I am getting the same reaction on the OM1 which is full manual. I can of course set the exposure manually. But is this something peculiar to Olympus and long lenses ?