Actually it's very simple. It's because a photograph is usually flat. Whatever angle of view was provided by the lens at the scene - the end result is still a 2 dimensional flat image.
If the image is very large, the viewer will tend to stand back from it, if it is very small they will move in very close. It is not dependent on the angle of view, if you use an ultrawide angle lens, but print very large, people are still going to stand back if they want to comfortably take in the whole scene. If you use a telephoto lens and print very small, people are going to move in closer. It's the size of the print really, not the angle of view.