Foam core sounds like a good idea. You may find useful materials at an art/craft store (like Micheal’s or Pat Catan's).

I made my first pinhole out of a cigarillo box from a cigarette store; it was 4x5. I drilled an hole in the lid and hot-glued a piece of flattened Pepsi can that was painted black, and made the hole in the can with an insulin needle from my father. Some balsa wood to make light-traps, glue, and black paint, and I was in business. It was quite cheap to make.

Like you, I want to make one that will take film holders (though I may stick to 4x5 for now).

This site has some useful information:
http://mrpinhole.com/

Don't get too fancy with your first attempt - you'll likely come up with design improvements after you use it a couple times.