It's been said that when using a brown toner it can result in a more fragile emulsion afterward, but I've never used any additional hardener and haven't ran into any issues. I don't throw my prints around but I don't baby them either.
The only brown toner issues I've run into were related to not using distilled water with Viradon. But this residue, if it even appears, is fairly easy to deal with after the fact.
Stop worrying about grain, resolution, sharpness, and everything else that doesn't have a damn thing to do with substance.