If you don't neutralise the developer carry over that will help take the fixer more alkali and effectively you'll have a more than a trace of development taking place as the fixer's beginning it's task of dissolving unexposed and undeveloped silver halides. This can result in dichroic fogging - a patchy stain.

It's even more likely to happen with prints with no stop bath whether acid or alkali/neutral fixers but at least with a print you can make another once a neg is ruined it's gone

Then there's the ammonia smell as well