Are you just pushing the pin through or using a piece of wood to back up the metal as you punch it? You should also deburr the hole very carefully with some very fine grit sandpaper.