The old vignette masks achieved this feathering just by cutting a jagged, toothed, edge instead of a smooth line, and since the mask is close to the lens and way outside the DOF range, it produced a softer effect. There are scissors available in craft shops that cut various kinds of jagged, deckeled, or toothed edges, so maybe there is one that might work well for this purpose.