In his book, Edge of Darkness, Barry Thornton did an experiment. He put his easel on a 1/2" thick piece of board then focussed normally and made a print. He then made a print without the 1/2" board and another print with two boards. He says that he could see no difference in the sharpness of the prints 1/2" either side of optimal focus - and he was a sharpness fanatic!
So basically, a 2mm difference will not be discernable - however, you do need to be focussed first, and this is where you are having problems.