A filter's effects will depend on the color sensitivity of the emulsion, as well as the colors of the subjects of the photo (including any colors caused by the color temperature of the light in which you are shooting).
However, VC printing paper is a special exception, as there is more going on. A VC printing paper has a high-contrast layer that is sensitive to one color, a low contrast layer that is sensitive to another, and the entire paper is not sensitive to red. There is a change in over-all contrast with a change in filtration. Specifically, the yellow filter would lower the over-all contrast, like Vaughn sez, and a magenta one would raise it. You do the same thing in the darkroom when you move to a lower filter number; you add more yellow filtration in relation to the magenta. This makes it so that the low-contrast layer of your VC paper gets more exposure than the high-contrast layer, causing it to dominate the picture more, resulting in lower over-all contrast.
With a panchromatic film (unlike photo paper in that it is not VC, and in that it is sensitive to all the primary colors of light), the filter will simply darken opposite colors and lighten its own in relation. The change is in how specific colors are rendered tonally, not in over-all contrast.
So, to answer your question, with a VC paper in the camera, the yellow filter will lower contrast in all situations. With a panchromatic film in the camera, the over-all contrast is not changed, but the apparent color sensitivity of the film is; this means a change in the tonal relationships that certain colors produce.
You can also use VC filters on your camera, of course. They are actually designed specifically for altering the contrast of photo paper, so you will get more predictable results with them as opposed to using your common Wratten-numbered filters.
IMO, it is better to think of filters' effects in terms of the colors of subjects, not in terms of specific subjects themselves. For instance, think of it as the yellow filter darkening things that are blue, as opposed to it darkening sky. Sometime the sky is more or less blue than at other times, and sometimes it isn't really blue at all (overcast days, warm light, etc.).