you aren't imagining anything -- a large format negative contains more information that your brain appreciates even if it is not immediately apparent to the eye -- enlarged small or medium format negatives have the same image, and maybe you don't see the grain, but your brain is filling in gaps left by smaller negatives when they are enlarged, while a large format negative has a lot few gaps and more information.