Is your pinhole in bare, reflective metal? If so, try "smoking" it with carbon from a candle flame or blacking it with a black permanent marking pen. Was the sun directly overhead? You may be seeing reflected light off the pinhole edges.

Lee