Seen from under an angle, not from straight behind, a circle appears as an ellips, is smaller.
Make the hole ellips shape, and from the side the long axis points to, the effect is negated. But only from that side, and only at a certain angle/distance from the image's centre.

I don't know how that would creat astigmatisme, except that the shape is echoed in the pencil of light's shape. But that's not astigmatism.