The washers behind the shutter are not critical to the functioning of the camera. As long as the film wind and shutter cocking works, I wouldn't worry. If any of the washers are shims for lens infinity setting, there are other ways to achieve this. Your primary issue is to get rid of the softness in the center of your images. The Xenotar should NOT be giving you images like your links.
Take out the washer and tighten the front lens group into place, finger tight is fine. No 'fraction' out or such. 0.5mm is not a small amount in optical systems!! Rollei's spec for alignment of the lens board to the film plane is 0.05mm (1/20thmm, one-half of one-tenth of a millimeter, .002 inch) maximum difference between the four corners of the film plane, and it is this large because you are dealing with depth of field and such. And this large a number assumes a precise optical/shutter assembly, which comes about from machine work, not random washers and shims.
Once you get the shutter block properly assembled and giving you relatively flat focus plane, correcting for infinity focus isn't hard, and getting the viewing and taking lens in alignment isn't hard. But none of that is worth much if the lens is giving you poor images.