I used to use a very cheap canvas bag with a pseudo military look to it. I stopped using it when I found a thin film of canvas dust all over my camera.
I generally dislike bags as I usually end up carrying the camera and an empty bag as separate items. I carry my Rolleicord on its strap on my shoulder and carry film, filters and lightmeter in jacket pockets.
The primary reason for using a bag, to me, is keeping organized. Take it out of the bag, put it back in the bag, and then it doesn't get lost or left behind somewhere and I don't have to do the self-strip-search trying to find where I put the meter/film/lenscap/etc. I'm one of those people who would lose his head if it wasn't attached, so it helps with a bit of extra discipline. It also helps provide some cover for my gear if I'm out in inclement weather. Although once I'm out and about photographing, like you, the camera is around my neck. But it makes for handy cover if you need to shield the camera from prying eyes while in transit through less-friendly areas, or if you're going in a museum or someplace where photography is not allowed.