I believe Adams had essentially the same observation you do. Gray cards don't necessarily get you a direct reading match to the middle gray or zone V that anybody might want, nor to an incident meter.

Reference cards of any color or shade, camera bags, blue jeans, whatever we choose to use, have to be placed or held at a proper angle and in the proper light and spot metered well to get close. Needing an offset from the spot/reflected meter reading to get camera setting is the norm even with a gray card.