The film plane should be at the same distance as the ground glass, shouldn't it? I don't have these problems you describe when going to 12:1 and beyond on 4x5".

I have found that loading the film holder into the back while it is still attached to the camera can cause a shift in focus because of the bump. As a solution I simply remove the entire back, load the holder, and gently return the back.