I guess somebody should turn me in, then... I took a photo of a federal building in Wilmington, NC last weekend.

I think the Federal Reserve Bank buildings do receive some extra security considerations, however I used to walk past the one in Charlotte everyday, almost always with a camera, and I am certain I've photographed that building (not that the photos were very impressive, it's just a big modern brick building...). Nobody ever approached me but this was before 9/11. I've checked online and so far I haven't found anything specific prohibiting photography of federal buildings.