The only thing with shooting at ISO 640 is that you lose some contrast? I really like shots that have that really 'deep' look to them. (full of color or tone) and I find that when I underexpose it becomes very air and light... almost kind of faded... especially in the shadows!
Pushing film should actually INCREASE contrast. What you observe might be cause by some different placement of luminance levels across the toe region, maybe.. That is, if you push a negative, shadows tend to be more "packed" and have less local contrast. If you or your printer try to raise them to brighter levels, you might end up with muddy shadows, which might be your case.
Just my 2 pennies from the small experience I have. :-)