It depends... I remember reading lens tests comparing lenses to their faster counterparts and at f8 some of the slower lenses were just as good or slightly sharper. But say going from a 2.8 vs a f4 or 5.6 there is a larger difference of sharpness in favor of the faster lens when shooting stopped down.

I personally favor using the faster lenses because of the latitude they afford me in various situations and the brighter image in viewfinder.