If you are interested in very smooth tonal gradations of grayscale, and if you are interested in as little grain as possible, you should carry the largest format camera around that you are able to carry, and you are able to use with as few limitations as possible to what you wish to achieve.
If you are interested in only getting a good picture, you carry around the camera that you are intimately familiar with, and don't give a dime whether there is grain in the picture or not, because you know that for most purposes, even 35mm will be more than good enough with super careful processing and printing.
Likely you're somewhere in between, or maybe even do both side by side. Then you have to compromise.
There IS a difference between 645 and 67. No question about it. You will get 'better' print quality from a 67 negative. The real question is how much that really matters to you, and whether you're prepared to only get half the photographs in the same amount of film, and if the extra bulk doesn't bother you.
You COULD purchase something like a Fuji or Mamiya rangefinder and shoot 6x7 with a camera that is very light, portable, and extremely capable.
Good luck with your choice. Whatever you pick, make sure it meets your demands for portability, picture quality, shooting speed, etc.