I will just choose a mid point wavelength, say 550.
This is right, because it's exactly what one does when one uses a regular color film with a pinhole camera (and does not notice any sharpness difference in different colors). And moreover, one has no concerns either when using B&W panchromatic film.

Then,, each film plain will be a diffrent distance from the single pinhole, so I will choose the longest for exposure calculation
This I don't quite understand : isn't your camera designed to have the same optical paths in the three colors ? if it's so with a lens (to get three views of the same size and same focusing) then it's so also with a pinhole ?