I wouldn't just look at the deep psychological level. There is probably a strongly ingrained cultural response that comes from our continuous exposure to brightly coloured advertisement, contrasted with many people's attempt at ennobling their subject by shooting in B&W.
What's the most common reaction I hear about B&W ? That it looks more serious, more grave, more intimate, etc. But does it have anything to do with fundamental vision, or is it more because a specific aesthetic has been pushed onto B&W once colour became readily available?