The only time I was challenged was in Stanley Park, Vancouver.

I'd gone to Stanley park with my Grandmother many times as a child. Back then it was a zoo (literally). So, when I returned for the first time in 20 years (in 2007), I walked around wondering where all the flamingos and other animals were. I eventually asked someone who told me the zoo aspect had shut down more than fifteen years before. And so I began to wander aimlessly, just walking the island to see what it was now. At one point I decided that "life is always more interesting off the main paths", so I should take the next small trail into the woods I could find. Sure enough, not 30 seconds later I saw man walk into a small but well worn path in some dense woods. I followed suit.
Inside the dense woods it opened up and there were many well worn paths crisscrossing. Benches had been cut into downed trees, and it looked really rather unexpected. And so, out came the camera (digital, all I had at the time, and probably for the best). I was focused on the paths and the trees, and rather oblivious to the people around me. I noticed a number of men walking about but didn't think anything about it, until one came up me and was visibly upset.

"If you take a picture of my I'm going to f***ing flip!" He said with a pronounced lisp.

I felt like a deer in the headlights. "Umm.. okay." I hadn't taken a picture of him, or anyone, just the tress and trails. I said as a much and showed him the pictures I'd taken. He stormed off leaving me utterly confused. At which point a kindly older gentleman walked up to me with a smile and asked if I knew where I was.
"I was going to say 'Stanley Park', but since you're asking me, I'm going to have to say: No, I don't know where I am."

"You're in the Gay pick-up area."

Right. Well. That explains a lot. I laughed good and hard at myself, thanked him for letting me know, and proceeded to put the camera away and leave. As I left the woods, still laughing at my own idiocy, I passed an elderly woman who gave me the most hateful glare. I laughed harder, found a bench to sit on while I typed out the story in a series of texts to my girl friend. She, not knowing how this was going to end, read them aloud to her entire office. It's been six years and I still haven't lived this one down.