Why does Scout attempt to engage Mr. Cunningham in conversation? Why does this exchange convince Mr. Cunningham and the other men to leave?

Scout talked to Mr Cunningham primarily because she is embarrassed and nervous. When she sees Atticus confronting the mob in front of the jailhouse, she does not know specifically what is wrong, but she instinctively wants to go to her father. In doing so, she finds herself in front of a crowd of unfamiliar faces and she is the center of attention—everyone is watching her.
It is a relief when she finds Mr. Cunningham in the crowd. Her talk with him is simply polite conversation, meant to cover her awkwardness, but his acknowledgement of her means that he can no longer fade into the crowd, and must take responsibility for being present. Scout's talk of his son humanizes him; there is also the sense that Scout is acting in the same way he would want his son to act in a similar situation. Scout's manners, in other words, are stronger than the hate of the mob and serve as a kind of unwitting admonition that they should behave better too.

An angry mob has descended on the jailhouse to lynch Tom Robinson. At the head of the lynching party is Walter Cunningham Sr., a well-known figure to Scout due to her being in the same class as Walter Cunningham Jr. She also, crucially, knows that Atticus does a lot of legal work for the Cunninghams. But as they're too dirt-poor to pay money for Atticus's legal services, they pay him in kind with hickory nuts, potatoes, and turnip greens.
Scout has learned well from her father the value of putting yourself in other people's shoes. As the situation outside the jailhouse threatens to get nasty, Scout reaches out to Mr. Cunningham, engaging him in conversation about his legal affairs. Mr. Cunningham is taken aback; he wasn't expecting a young girl to start chatting with him about the finer points of entailments. Scout also asks Mr. Cunningham to say hi to Walter for her.
Immediately, Scout's unexpected intervention has taken the heat out of an ugly situation. By establishing a personal connection with Mr. Cunningham, Scout has reminded him that he's a father too and that it would be wrong to hurt a father in front of his child. In other words, Scout has passed on to Mr. Cunningham the lessons in empathy she's learned so well from Atticus.