Another thing to think about...if (when) you crash, would you rather have a hard object in a pack on your back when you land, or have that hard object in a pannier. And where would it best for the camera to be in that situation?

The two times I have crashed, I have rolled and have been able to reduce the damage to my body -- I don't think I would have done as well with a pack on with a camera body in it.