I don't think either is more difficult than the other, but they are very different. For what HCB did, black and white suited his vision. The same was true for very different reasons of Ansel Adams who did dabble in color but whose color work was, IMHO, pretty unremarkable.

Kerry Thalmann is one photographer who comes to mind whose work I like but just wouldn't work well, most of it, in black and white:

http://www.thalmann.com/