A lot of this has to do with CYA. Police, security gaurds and building officials probably do not know the specific rights of the public with regards to photography. Better to stop the photographer and not get into trouble and lose your job if they think it is illeagal to photograph the building. I don't think it is illeagal to photograph anything from a public right of way. On the specific site is a different matter.

Pre-9/11 I was with a friend on a county road just off the property of Offutt Air Force Base in Bellvue, NE. We were photographing the comings and going of aircraft during an airshow. A vehicle showed up with two base security officers. They asked to look at our cameras and in our vehicles. We complied and after deciding we were pretty harmless, told us to have fun and left. I assume the outcome would be different today so I have not been back.