I agree that most photography isn't or doesn't try to be art, but it seemed to me his opinion was that photography made with artistic intent, is most often only craft. Which is a belief rooted in ignorance and limited understanding of creative photography. Which is what we've been talking about.
That's exactly what fine art photography intends to be, decoration. As opposed to simply art photography, which carries messages, symbolism and is made from an intellectual standpoint. This is the work people need to be looking at to gain a better understanding of the term 'fine art'. Even Ansel Adams had a problem with the term for the very reason that it had historical connotations in the broader arts (as well as photography even then) with mere decoration, for which his work was often criticised. Whereas Weston's work was further removed from the superficiality of 'fine art' because of its connections with modernism. I think both of them would be turning in their graves had they known such discussions were still going on.
So I assume that many photographers produce work that could be considered fine art and/or art? Perhaps putting a label on anything like this is not a good idea.