Why not have 3 pinholes very close to each other (optimized in all respects for each color) - L and R (top/bottom whatever) reflected 90deg away from the centre which is left unaffected ...

Parallax issues could be smaller than the effects of the pinhole size/wavelength issues ?