IMO, there is no definitive answer to your question.
I think, at least for me, it is a matter of my own sensitivity to the "tonality" found in black and white. When I see a b&w photograh for the first time, I will always first notice the tones and how they interact with each other before I examine the actual subject matter. It may sound weird but that is just what happens for me, so it is in my reaction to the tonal relationships that are occurring.
For me, with color, there is no such reaction that forms within me. It simply is what it is and I am not free to ponder it for any other possibilities, rather I am forced to see exactly what it is.
As I read what I just wrote, I am not so sure that will make any sense to anyone but me. Just my 2 cents.