I understand your feeling. Part of my affinity for Panatomic-X is the recommended Fix time of 2-4 minutes, compared to 5-10 minutes for other film.
Completely trivial but adds to the pleasure of working in the darkroom.
Now that I am more results-oriented (develop to contrast index, fix twice the time to clear), the difference is even less meaningful.
And of course, now that I am hooked completely, a few minutes extra processing time has no bearing on anything.