It really matters who the portrait is for, too. I'm sure that many photographs that we see in museums, by famous photographers, are commissioned works. Somebody wanted a photograph, and the photographer hired tried to appease.
A portrait of a young child, as a gift to the child's mother, would be approached in an entirely different way than say a portrait for a TV documentary about World war 2. Two completely different aims, but without knowing that we'd look at them with the same pair of eyes.
To me, the appreciation of a portrait is an individual journey. Some speak to me and others don't. Others might have the exact opposite experience. One man's ceiling is another man's floor, according to Paul Simon, and I tend to agree. We don't all appreciate the same things, hence it is, again, a matter of subjectivity.
That is, unless you have trained to look at works of art objectively, which is a very different skill, but that's more of a value assessment than an emotional journey.