I'm not a pinholer, but as an engineer I think simple optics is the problem.
With a 1.2mm pinhole diameter, even a subject at infinity will only resolve to a 1.2mm diameter circle on film. This gives you a resolution of about 30 x 24 on 135 film. Closer subjects will reduce resolution further. In order to improve resolution, you need to reduce the diameter of the pinhole; but at some point you will run into problems with diffraction, since the long focal length will magnify the ray divergence due to diffraction.
 OK, I did the maths and I think you will become diffraction limited with a pinhole diameter of around 0.4 mm. At this point, the pinhole diameter is the same as the airy disk diameter and the total uncertainty (calculated as the RMS of the airy disk and pinhole diameters) is minimum at 0.55mm. That still only gives you resolution of 66 x 44 or so on 135.
Calculations were for green light (500 nm) but blue light doesn't gain you much - it only reduces the RMS circle of confusion to 0.52mm.
So the problem with long focal lengths is that the f-number is small (f/625 in our example with a 0.4mm pinhole) which puts a lower limit on pinhole size (due to diffraction) that is still too large for good resolution (due to geometric optics). This is a fundamental physical problem, nothing you can do about it (except to reduce your pinhole size to 0.4mm to get the best possible resolution at that focal length).
Yoda says "Obvious the solution is. Bigger film you must use." Many years ago when I was in High School I built a 4x5 pinhole camera for shop class. A couple years ago I built a 35mm populist pinhole camera. I got better results with the 4x5 than I did with the 35mm. I think andrew.roos calculations are valid. If the optimum pinhole diameter is 0.4mm and you get 66x44 or so on 135, then by enlarging it 4x to get a 4x6 print your COC is far too large at 1.6mm. If you're using 120 in 6x9 format your resolution with a 0.4mm pinhole becomes roughly 150x225. Enlarging to a 4x6 print gets you a magnification factor of 1.69, so your 0.4mm resolution on the film gives you .68mm on the print. You'll get even better resolution on the 4x5 with a 0.4mm pinhole (something like 254x317), and since the film is so big you just do a contact print, which will look even better than 6x9 on 120, and completely blows 35mm out of the water.
Shoot more film.
There are eight ways to put a slide into a projector tray. Seven of them are wrong.