Ken, I think I agree at to some degree. It pares down the experience and as a result we look at things in terms of lightness and texture. It's not that we don't do that with color images -- but we don't do it the same way. Of course color can communicate a great deal. Think about Galen Rowell's stunning photo of the rainbow over the Potala Palace in Tibet. It would take an utterly virtuoso effort for that photo to work in B&W.
If you look at Galen Rowell's landscapes of Yosemite and the southern Sierras, which in my mind are every bit as great as Ansel Adams', you can see that he was very much concerned with light and texture in a color medium -- and to tremendous effect. One difference between him and Adams is that a great deal of Adams' effectiveness had to do with using filters, expanding and contracting development, and dodging and burning. On the other hand, as a 35mm slide film user, Rowell's approach was one of being in the perfect place at the perfect time -- something a lot easier to do with a 35mm Nikon.