I have found that color mats seem to compete with the BW like oil and water.

On the the other hand I knew a lady whould hand color only one very small part of her BW images. She would mat the picture in a mat very close to or contrasting with that one color and it was really neat.