I find that smiling innocently and saying "I didn't know that!" while continuing to focus. Then "Do you know the phone number of the security office/police in charge?" while setting your exposure and adding "I certainly don't want to get in trouble." while putting film holder in and saying "Thank you for that information." after hitting the shutter release usually works.

I had a similar situation near a bridge in the port. Got stopped by port authority police, building maintenance/security manager (who shoots his cameras regularly around the area), and a security guard. I find that being calm and forthright (heck, I'll let them have my ID) calms them down as well. I don't mind being stopped, because it means they're doing their job. Is it a hassle? You bet. I'd be more worried if they didn't talk to me...