I've run into the same stuff a few times.
The most dangerous are when you're out at night, that's when peoples' fears really get wound up. I've veen out shooting old buildings at night and have had people come out to run me off, treaten to call the cops, etc. I just go ahead and take my picture and move on. One night in Mexico I was charged by a guy with a 3-foot machete painted red on the end. I passed up that photo.
Once in a county fair during the day, I was shooting crowd pictures (got a few intereting ones, too) and as a group of people were going by, I snapped the shutter (with a Leica M6, of all things). A guy somehow heard it and charged back at me accusing me of taking a picture of his girlfriend's rear-end. I was getting ready to get in a fight - I don't run unless you have a machete, that's my policy.
So, you run into all types who have all sorts of different fears and points of view, etc. You'll get over it.
I agree you shouldn't take be business card. You don't know who she has hiding in the attic.