Yes it does matter. As some woods age they warp. Some woods are oily and who knows what that would do to your negs. Plywood is always an option but other woods just plain look good.
You can make a camera out of plywood and it would serve our purposes but people don't.