I'm not a fan of epoxies. They tend to get brittle when they're cured.
Gorilla glue works well but I seem to get a residue on my delicate digits.
Weather strip cement from the auto parts joint works very well. It's made to cement rubber to metal.
how about some sort of thin double sided self adhesive foam? I'm sure you could find something like that at a craft store. Same idea as the felt, is that its removable if you decide to change the focal length.
There were a lot of answers to think about. This weekend I was able to use my gf's uncle's bench press and tools this holiday weekend when we went out of town. I was able drill the metal very nicely. And after checking the fit and a test exposure, I decided that it actually does a nice job baffling the light already. I was worried that light would seep in. So since its doing pretty well already and there isn't any true wood/glue solution, I decided not put any kind of glue. I'd rather not make things messier if I didn't have to. the fit is very snug, but since I'm anal, I'm going to thinly gaffer the inside. I appreciate all your input. Its probably better for me to just keep it simple