Why was Peggotty annoyed if she caught David looking around?

Peggotty is the servant who helps care for the young David Copperfield. He, his mother, and Peggotty attend church together. Peggotty wants David to keep his eyes on the preacher and not let them wander all around the room, though she herself sometimes lets her eyes wander. The implication is that it is respectful to gaze at the pastor and pay attention to what he says.
David, nevertheless, is a young child, and his family knows the pastor socially, so it is strange for David to stare at him. He is afraid it might be rude to do so. Since he's so young, he also has a short attention span, and it is difficult for him to keep his mind from wandering. For example, he thinks it would be more fun to throw cushions at his friend from the pulpit than listen to the sermon. Finally he falls asleep and falls off the pew with a crash.