With architectural photography, I've waited for aaages for a scene to be clear to photograph, whilst the light changes and I freeze or boil to death. I think in that circumstance, if the shot were critical and you looked like you were settled in, I'd come over and have a chat and try to "engage" you with what I was up to - discuss it like a human being (you may have more 'right' to be there than I do - who knows 'till there's a dialogue?). I reckon if I was you in your situation, I'd be grumpy too!!