What annoys me sometimes is people not paying any attention to anything what-so-ever, talking on cell phone, and walk really slowly through my frame or stop right in the middle. Usually, I just wait or move to a new location. I'm usually not in rush to complete my photograph.
Try walking right up to them and shooting closeups. If they complain, just say innocently, "Oh! I thought you wanted to be in my picture!
But how about walking through other people's shots?
In former times I always stopped when I realised someone was trying to take a photograph of some object/person and I was crossing the angle of view (asserting that only a rare breed of photographers would like me to interfere with their image).
With all that constant photographing all around I don't bother with any kind of photographer any longer.
(Might do so though with an analogue photographer, but don't come across such any longer anyway.)
The subject of the photograph dictates whether I stop or walk through. If someone is photographing other people who are obviously posing for the picture, they don't want me in the frame, so I stop, or walk behind the photographer. If someone is doing street photography, I just keep on moving, assuming that "street photography"implies the presence of "street people."
But the question does remind me of a humorous anecdote: I was making long exposures of a Great Lakes freighter unloading aggregate one night. A dock worker walked up and then stopped so as not to get in my picture. I told him that as long as he kept moving he would not appear in my picture, which of course absolutely astounded him.
I usually just smile and wave 'em on. The people that want to talk can be annoying, but I consider that I am a freakshow hiding under a big cape like batman or something. If I am in a hurry, I speak to them in another language. That always works to dispatch them quickly. Swedish is my preferable language for this since no one speaks it, but I like Swedes so if someone spoke back I wouldn't mind missing the shot in order to meet a Swede. Never has happened though.
I wave people through and, if they insist on waiting, I thank them and take the shot.
Once, I was walking in New Orleans' French Quarter and this fellow was getting ready to take a photo. I stopped and waited... and waited... and waited. After a couple of minutes, I started walking and he snapped the photo with me in it. He called me a turkey.