Perhaps the issue is caused by the change in depth of field and the thickness of the image layer. At f5.6, you might be focusing on the (for example) surface of the image layer, whereas at f11, you might be focusing on the whole image layer. The following article emphazises that instead of focusing on 'grain', you should focus to maximize image contrast - to quote from page 3 of the article:
"Good focus is achieved by finding the highest degree of contrast difference between light and dark, in a bit of blurry detail, while avoiding the use of film grain."