Take a look at the current Canon EOS ads with the members of the VII agency. These guys are all "in uniform" and no vests -- just generic urban safari-ish gear (takes a beating, big pockets, easy to wash). The vest screams "official photographer over here!" which is usually not a good thing in my book.
I have an anorak I bought at Eddie Bauer for $40. It's my standard rain jacket and travel coat. I can carry my Contax, three lenses, 8 rolls of film, a strobe and sometimes an EOS body in the pockets. Plus my wallet. It's a good middle weight for both summer and winter (with sweaters) -- I've worn it in July in LA (it's too big, I admit -- but that gives it breathing space) and in bitter Swedish/Minnesotan winters (with a sweater or two). Even wore it in Hawaii from time to time.
The only time I think I would want a vest would be at weddings -- the black ones would function well under a suit jacket.