Cool project, for fun at least!

I don't know too much about the electronics, but have some ideas about the other parts. I'm sure you have an IR issue, simply because you didn't use a filter. If trying to scavenge a filter, not only digital cameras, but any color video camera should also have an IR cut filter in it. Although it's probably part of a blur filter package, you can probably just use the whole thing without a problem.

In the meantime, here's another idea. If you have an IR-pass filter (visually opaque), you could take two readings, with and without filter. The difference between "everything" and "IR only" = visible light. You don't have such a filter? What about several layers of processed leader (opaque) from color neg film? For some reason, the dyes used in such film (or E6 slide film) don't seem to be able to block into the IR range, so they would probably serve as decent IR pass filters.
If you wanted to check on this, first see if you can read the output of an IR remote control. If you are able to do so, then block it with the dark color neg film and read again. I doubt that the black film will affect the reading, at least more than about 10% or so.

For another area to play with, I think you'll find that both fluorescent lights and LED flashlights have virtually all of their output in the visible range. If so, the black color neg film should stop virtually all their light (with respect to what your meter can see).

Maybe there's potential application for the people shooting IR film; it seems to be something of a problem knowing how much exposure to use.