May I just point out that paper is not panchromatic. The use of a dark orange filter will be turning your exposure in to a sort of semi-safelight exposure, and also reducing the transmitted green light used for the low-contrast multigrade exposure (as well as the blue light too of course). Also note that daylight is relatively blue compared to the light temperature coming out of the enlarger (for which the papers are designed) and that it is the additive effect of the blue-sensitive parts of the paper emulsions which produce the high-contrast, so it has a potentially larger share of the exposure in an unfiltered pinhole camera - to be clearer, you don't end up with a grade-2 middling contrast grade if you use no filter in daylight. Precisely what you do end up with depends a bit on what paper you have, the time of day etc.
Assuming the paper is multigrade, then use of yellow-ish filtration might help control the contrast slightly, at the expense of more exposure. You can also try a pre-flash, to raise the shadow exposure a little, and then adjust exposure to suit your highlights.
Check that your pinhole is the "optimum" size (should that really be f344?) and then base your exposure on that value, a light reading and the Ilford-Harman pinhole exposure slide-rule.