even the math and science is often influenced by such things (the Newton point is just one such example - in a way we still have those two colours as an "official" part of the spectrum).

Colour often doesn't even transfer from one culture to another. What someone in Japan calls green you might call yellow. Some languages lack words for colours you or I "see" - others have words for for colours you or I don't quite perceive.

Evans and Evans, Hanson and Brewer are both very good examples of this. Plenty of math and scence with what you might call a nice mix of psychology and culturally relative conditioning

The math and science is only one part of it. How we actually perceive and understand colour is another. For example, the whole colour wheel/complimentary colours idea is similarly very much a social construct.