tkyama is correct; it is voltage under load that tells you the condition.
electrical engineer's explanation (optional):
The simplest model for a battery is an emf (from the chemical process) in series with a resistance. As the battery fails the emf stays the same but the resistance increases. Hence as you draw current the voltage observed at the terminals of an exhausted battery will be less than that which will be produced by a good battery because of voltage drop across the internal resistance.
quick and dirty test of battery condition:
If your multimeter has a 10 amp dc range (be sure you plug the leads into the right places!!) connect it briefly (emphasis on briefly) across the the battery terminals and measure the short circuit current. The difference between a good battery and an exhausted one will be quite apparent.