I have both Red and other coloured filters together with a Polariser.

The effects are different.

As many of the previous posts have indicated you need to try them out to see how they affect your sort of photography.

The weather, atmospheric conditions and subject matter all have major influences on the effects you get – so unfortunately there is no one size fits all rule – you have to work it out for yourself.

If you are going to do more B&W work, then you might quickly find that only having a Red coloured filter is rather limiting – Red can be a very harsh, and you might need a range of filter colours to give more subtle effects.

Personally, I only shoot B&W and have over the years built up a range of 7 different coloured filters plus a Polariser for each of the filter sizes I have.
90% of my work can be covered with a medium yellow, a medium orange and a yellow/green filter.

Of all the Filters I own the Red filter and the Polariser are the least used – but it’s a very personal choice – YMMV

Good luck