It might be ingredients (chemicals) in the clay or the polyurethane that are fogging the LL, although your problem does not sound like that. If chemical fog was taking place, it would be constant regardless of exposure.
OTOH, you may be washing something out of the clay as you repeatedly recoat it with LL and this might be decreasing the chemical fog. IDK.
It certainly is odd. But, the fact that it was not a smooth surface is not part of the problem.
I'll give it more thought.