In "The Custom House: Introductory," what is Hawthorne's attitude towards the community of Salem?

Toward Salem itself, the narrator of this introduction, who seems to be Nathaniel Hawthorne, professes to have something like affection, despite the fact that the town seems rather haphazardly planned and its people leave something to be desired. He believes this good feeling toward the place probably arises from the fact that his ancestors came to Salem so long ago, so he has roots there.
Hawthorne seems to find some of his peers to be in their prime and interesting, but the vast majority as "wearisome old souls, who had gathered nothing worth preservation from their varied experience of life." Near the end of the introduction, he says the "good townspeople will not much regret me," and it seems like he finds them somewhat small—perhaps a bit provincial. While they toil away, he is drawn to write this story, a story he seems to believe they cannot appreciate; Salem is not the "genial atmosphere which a literary man requires, in order to ripen the best harvest of his mind." Thus, he feels relatively unneeded by the people of Salem (as he's lost his job anyway), and admits the people of Salem hold him back intellectually anyway.