Base your "push" or "do not push" decision as much on the inherent contrast of the scene and the light as on the light levels.

A push increases contrast, but doesn't increase sensitivity (much).