Imagine Rubylith is a Jello-like material on clear film base. You cut the softer red mask material with a sharp knife without going through the base. Then you peel off the ruby oval, leaving the oval mask with clear film in the middle.
I don't think you would want to enlarge through this (the film base would degrade image quality), but if you were making contact prints, you could create a sandwich with the Rubylith mask on top of the negative.
Something new that might be applied to the task: You could use a Cricut machine to cut through and make a perfect shape.