My personal maximum is 16x20, for the same reason as DrPhil. 4x off a 4x5 negative is about as much grain as I can handle. However, I know of one photographer who goes *large* with an occasional 4x5 image, and it looks just fine; exceptional, in fact. Yes, there's grain when you stick your nose in the print, but so what? Step back and view the entire image, and the grain goes away.
So it is a personal preference.
Some images are strongest when printed a certain size. For example, a photograph with areas of very little texture, such as bright snow, or very deep shadows will work fine when printed on a smallish paper. However, go larger and these small areas become large areas of nothing-ness that will detract from the overall image.
Hope that helps!