There are many ways that masking films or masks can be used.
1. Color correction masking. A mask or masks are used with transparency film to correct color errors in a manner similar to the orange mask that is present in color negatives.
2. Unsharp masking used to change the 'tone' of a print from a transparency. It is similar to using a pyro developer with B&W.
3. Contrast control. This is a bit different than unsharp masking in that it is used to lower the overall contrast of a color transparency and make the print less garish, the usual result from pos-pos printing. It affects the entire gamut of the transparency, but #2 and #4 below affect mainly the toe of the final image.
4. Highlight masking. This is similar to #2 above but can be applied to any color record or to all 3 in a given scene. Often 2 and 4 are lumped together into one subject, but there are subtle differences. For example, the highlight mask can be sharp, but the unsharp mask is by definition unsharp.
Kodak has published a series of dataguides on all of these topics, but since the films are no longer available I doubt if the books are in print. In any event, if you can get them I suggeste you read up on it. They also have some very good photographs illustrating the results from each method. They also include some of this in their dataguides for the printing industry where they discuss halftone printing of color.