There are two causes for an image circle that's too small for your film format. The first one is something blocking the light... that usually gives a very clean circle (with sharp edges) it's quite possible that the shutter opening in a brownie is too small. You have to trace an imaginary line from the corner of the film, through the pinhole and out again... is there any piece of the camera in the way? then you need to remove that or find another camera. I don't know about the brownie, but many box cameras have very tight shutters that can give these problems when converting them to pinhole.
The other possibility is wrong material for the hole. If the material is too thick, you get more light fall-off at the corners. If your image seems relatively normal but gradually becomes dimmer at the edges, that's probably your problem. To avoid this, I use regular aluminium foil from the kitchen for my pinholes. It's not very robust, but thin enough for even the widest angles of view.