Maybe most of my problem is this: the patch image is always higher than what I'm trying to line it up with. It's not much but it's definitely noticeable. And I know it's not correcting itself as I go over the object because I can clearly see it start on one side of the object slightly higher and if I move the focus ring so that the patch image crosses over and goes to the other side of it, it's still slightly higher.

That makes focusing difficult because even when things are lined up, they often look slightly soft.