I got harassed for pointing my camera at a bank in Pasadena last summer. The man (an employee of the bank, dressed like a middle manager) asserted that it was "illegal to photograph financial institutions", even from the public sidewalk I was standing on. He hurried off to tell the police, and I walked away. I kind of wish I had stayed and argued for my rights, but I didn't.
In my experience, most people are turd burglars.