Compose as much as you possibly can before you even take up the camera, shoot quick, walk along as nothing has happened. Most people do not notice until after you've shot, if at all, unless you stay and fiddle around for too long.
I live in France, here it's illegal to use pictures with peoples faces in public without their explicit permission.... Lot's of people seems to know about this, so the only way to go here, unless you want pictures of people posing, is to be fast and discrete.
I wear a hat, sunglasses, and have my 35mm SLR around my neck hanging down in front of me so I look like a tourist. Perfect disguise... (plus that I am way taller than any french person I've ever met, and blond (scandinavian) so I really look like I dont belong here...).