If you were not on government property, he can't tell you to do anything. Even if you were on USPS (which is only quasi-governmental, BTW) property I would question his jurisdiction unless he's part of the Federal Protective Service, which provides security for Federal Buildings.
Originally Posted by photomc
I work at a US Government facility. If we observe a breach of security we call the FPS. I'm a contractor, but not even the Fed employees can take any action on their own. Unless I'm challenged by a uniformed officer with a badge, I'm very uncooperative.
I would've continued photographing and told the man to kindly bugger off.