As Jim Chinn pointed out, understanding the US and it's current context is key. There has been endless conversation within this country after 9-11 requesting that ordinary citizens "be vigilent" and to report suspicious activity. While it may be reasonable to question the wisdom of this, consider that the terrorists who attacked us on 9-11 did so from among us. In the parlance of the intelligence community, they hid in plain sight.
We are a very independent people by culture. Generally, it is not our government imposing restrictions (there are some new impositions, but let's not carry this discussion in that direction please), it is ordinary citizens trying to "do their part" for homeland security. It is certainly the case that many will go beyond what they have a right to do, and will infringe on the rights of others.
If you are visiting from a foreign country, I think it is prudent to be sensitive to the cumulative impacts of the 1993 WTC bombing, the Oklahoma City bombing, 9-11, and the nearly-successful Millineum Attack (a foiled attempt to attack LAX). Add to that current news that bin Laden is directing Zarqawi to attack the US directly, and constant reports regarding the porosity of our southern border, and you might understand why ordinary people are more wary than they used to be.
Our government, rather than taking the unacceptable step of locking down public places (for the most part), has asked its citizenry to "be vigilant". Unfortunately, sometimes these citizens decide to "be vigilantes".
I'm not defending behavior that infringes on your rights or my rights. I do believe it is prudent not to be grumpy, confrontational or defensive in those circumstances. I think it is quite reasonable to smile, offer the person a copy of that nice document linked to above regarding your rights as a photographer, and be otherwise civil. (And after the fact, if you are so inclined, sue the bastard! ).
Our current culture (one of a new vigilance), customs, and laws may be more restricted than those of your home country. That doesn't make the US bad, just different. Be sensitive to that, and be prudent.
I think it is just as prudent for US citizens going abroad to be just as sensitive to the local laws, customs and culture. Many (not all, or even most, but some) are more restrictive than that to which we are accustomed. Some are less restrictive. In those cases, I say enjoy, within reason!