Okay, so I have and (more or less) regularly use the ZF.2 50/1.4, AF-D 50/1.4, and Ai 50/2. Among them, I like the ZF the most, and the AF-D the worst. Actually, my copy of the AF-D suffers from some "blue dot" phenomenon, where under certain (outdoor) lighting conditions and apertures a blueish circular haze can be seen in the image center. This is most noticeable on digital sensors, but can sometimes also be seen on color film. This makes this lens pretty much useless for me, except for indoors use.
In a side-by-side comparison at infinity, the ZF clearly beats the AF-D at all apertures up to f/8. The image borders and corners are noticeably worse on the AF-D until f/2.8. However, the ZF shows some "haze" due to residual spherical aberration, i.e. the sharp image is overlaid with some hazy shine. This is gone by f/2.8.
In the same comparison, the Ai 50/2 is about on par with the ZF from f/2.8 on, and sharpness wise slightly better at f/2 due to the lower amount of spherical aberration.
However, where the ZF.2 really shines is the range f/2.8-f/5.6 at medium distances, where you'll get a beautiful "popping" rendering emphasizing the subject's 3-dimensional structure. Here, the Nikkors just cannot compete IMHO. And, this is not an artifact of digitally emphasizing image sharpness - it works on Rollei Retro 100 with Rodinal as well (or maybe even better). On the other hand, the ZF.2 is not a good available-light lens for wide open night time use - lots of coma make it practically impossible to correctly render point-light sources wide open.
The ZF.2 50/1.4 is my only Zeiss lens for 35mm. I got it, since the performance of the AF-D was so much hampered by the blue dot. If I hadn't lots of good choices in other focal lengths, I'd probably add one or two more ZF lenses at other focal lengths, however, given their price point, it's hard to justify adding one that's redundant to my other glass, as long as that performs "good enough".
So, yes, I think the ZFs are significantly different from the Nikkors, but whether they're better depends on ones usage.
As for the ZFs being MF only, yeah, that's a pity to some extend - most modern focusing screens (post-AF era) don't allow for proper manual focusing. On the F5 it's no problem to change the screen against an A type, for example, but on the digital bodies proper manual focusing is pretty difficult. I'm still thinking of filling my 85mm-portrait-lens-gap with a ZF.2 85/1.4, but the difficulty to properly MF this on some of my bodies makes this decision pretty difficult...