I'm 1000+ miles away from your area but had the good fortune a few years ago to be in that part of the continent for fall. My suggestion, go North young man. The Eastern Townships area of Quebec province have the same mountains, climate and trees as NH & VT, but nowhere near the crowds of sightseeing buses.
Brome, Magog and Greater Sherbrooke were the areas we visited and were very impressed.