You're sure the lens is fully extended and locked in place?
Does the distance scale on the lens roughly marry up to the range finder?
I have a very hazy memory of reading somewhere that at f/3.5 the GF670 lens is intended to provide a slight soft image intended for portraits. I can't remember where I read this, but I'm pretty sure I've read that the lens will provide a less 'clinally sharp' image wide open. If it's sharpened up completely by f/5.6, then it may be working as intended.