Why does Armand believe that Desiree is the reason the child is not white?

It seems that Armand Aubigny believes that his wife, Desiree, is the reason their baby is not white because her history is unknown. Desiree had been a foundling, encountered by Monsieur Valmonde, the man who would become her adoptive father, asleep at the gateway to his property. She was of a "toddling age," and when she awoke, she began to cry for her "'Dada.'" The narrator says that most people believed she'd been left, on purpose, by a group from Texas which had been passing through earlier that day. Rather than investigate Desiree's provenance, however, Valmonde's wife decided that Desiree must have been "sent to her by a beneficent Providence to be the child of her affection" since she was unable to bear any biological children. They, therefore, raise the girl as their own daughter from that point on.
It is possible, however, the Armand knows that it is he who is the cause of his child's dark skin color. In the end, we see him burn a letter written to his father by his mother, a black woman. It is unclear whether or not he knew about that letter before he threw Desiree out or if he only found it when he was cleaning out a drawer.