What is the town's opinion of Mr. Hooper at the end of "The Minister's Black Veil" by Nathaniel Hawthorne?

In the end of "The Minister's Black Veil," the townspeople fear the Reverend Mr. Hooper because of the ambiguity of secret sin and sorrow which envelops him with his refusal to remove his veil.
Those who attend Mr. Hooper's final moments "shrink from one another in mutual fright" as the minister refuses to remove his veil even as he dies. As he falls back, he utters these dreadful words,

Why do you tremble at me alone?... Tremble also at each other! ...when man does not vainly shrink from the eye of his Creator, loathsomely treasuring up the secret of his sin.

As he dies, Mr. Hooper tells those present, "I look around me, and lo! on every visage a Black Veil!"
Because they have been unable to see Mr. Hooper's face once the minister has donned the black veil, the congregation is unsure of how to perceive Mr. Hooper and the shadow he declares that he senses. Apprehensive about what the minister may perceive in them, people have avoided the minister, except for the occasions of certain services he renders as a clergyman because he is still perceived as very devoted to God:

in this manner Mr. Hooper spent a long life, irreproachable in outward act, yet shrouded in dismal suspicions; kind and loving, though unloved, and dimly feared; a man apart from men.

This description fits the town's opinion of Mr. Hooper at the end of the story.