Check that the adapter is properly fitted?
I thought the Adaptall mount was purely a mechanical coupling (no glass), so it shouldn't make any difference which is fitted. However, if not quite correct then it could put infinity focus out.
Are the pictures sharp? I've never really been able to spot a sharp lens through the viewfinder, only by taking a magnifier to the negatives afterwards.