In "The Minister's Black Veil," why do the villagers bury Mr. Hooper without removing the veil?

When the Reverend Mr. Clark sits by Father Hooper's deathbed, Mr. Clark asks Father Hooper to allow him to remove the veil from Father Hooper's face so that he can meet eternity without it.  Father Hooper, however, is horrified by this idea, and he shrieks, "'Never! [...].  On earth, never!'"  It is clear that Father Hooper wishes never to have the veil removed, and this may be one reason that his parishioners do not remove it.
Further, everyone gathered at Father Hooper's deathbed still seems to fear the mysterious veil.  Mr. Clark even suggests that it signifies some terrible sin Father Hooper had committed.  However, Father Hooper asks, 

"Why do you tremble at me alone? [....] Tremble also at each other!  Have men avoided me, and women shown no pity, and children screamed and fled, only for my black veil?  What, but the mystery which it obscurely typifies, has made this piece of crape so awful?  When the friend shows his inmost heart to his friend; the lover to his best beloved; when man does not vainly shrink from the eye of his Creator, loathsomely treasuring up the secret of his sin; then deem me a monster, for the symbol beneath which I have lived, and die!  I look around me, and, lo! on every visage a Black Veil!"

It seems to be, in part, this "mystery" that Father Hooper describes that so terrifies the people.  They seem always to have had an obscure idea of what the veil symbolizes -- that each of us has secret sins that we attempt to hide from each other, ourselves, and even God -- but no one wants to admit that they might understand because that would be tantamount to admitting that one has these secret sins on one's soul, and what everyone wants the most is to hide this very fact.  If, in truth, these people really do have even a vague understanding of the veil's meaning, then they would not want to remove it after Father Hooper has died because it is a meaningful and accurate symbol; if, on the other hand, they really don't have a concept of the veil's meaning, then they would not want to remove it because it is such a mystery, and we fear mysteries.