Folks, having looked it over many times, I finally decided to go the drill bit route.

I rotated the drill bit by hand, removing a little material at a time. I've left it just a little tight, to avoid any light-leak issues and keep the door firmly shut, but it's comfortable now. I'll clean up the felt with some scotch tape, in case some of the particles found their way to the trap. The hole in the door still appears circular to the eye, i.e., the eccentricity introduced is very slight.

Your replies are much appreciated. I was pleasantly surprised by how well you understood this physical issue, despite the constraints of the internet. Thanks!