Pretend it's suspended from a string.

Assuming a normal orientation for the camera - ie pointed horizontally, put the grey card vertical and parallel to the front of the lens (like a filter would be). Now rotate it vertically so it's halfway between the sun and the camera.

I think it's only a real problem with intense sources like the sun. If it's overcast or indoors with indirect light, it won't matter because there's no strong reflection from the surface of the card.