I've also often wondered why this book is so special to me. Except for a handfull of images they do seem pretty ordinary, and there is no direct correlation between any image and Nancy Newhall's essays (which are absolutely superb).
But it does speak to me as no other classic book of photographs does (and I am from the rural South). I cannot explain the attraction.
I gave my first copy away to a painter friend from the Cape, and went for years cursing myself before finally replacing it.
I think that if you don't appreciate it now, that's okay. Just keep it in mind to try again in 10 or 15 years. There really is something special about it!
Also, be sure that you were looking at the later (Aperture) edition, as the reproduction of the earlier (original) edition left a lot to be desired.