What is Shakespeare's art of characterization?

Shakespeare’s art of characterization—his power of creating personality through diction, revealing psychology through the words his characters speak and think—is the essence of his genius, and the reason for his enduring importance. As if often observed, the narratives of Shakespeare’s plays are almost all adapted from prior sources. His great originality consists in the richly textured and verbally exuberant characters, major and minor, with which he populates those familiar stories.
The most important early step in developing an appreciation for Shakespeare is acquiring sufficient confidence in understanding his language and its meanings that one is able to distinguish between the diverse voices of his characters. For many  students, unfamiliar with Elizabethan/Jacobean vocabulary and the rhythms of iambic pentameter speech, all the characters might sound as if they speak in an identically “old-fashioned” or “elevated” idiom. But once reading or hearing Shakespeare begins to feel more natural, one finds that Claudius sounds vastly different from Gertrude, Horatio from Polonius, Ophelia from the gravedigger—and all of them are light-years removed from Hamlet himself, who is probably the most searching, penetrating, and sophisticated intellect that Shakespeare ever created.
Entire libraries have been written about Shakespeare’s art of characterization, so no comprehensive answer to your question is possible. But I’ll direct you to two critical viewpoints that may be suggestive and helpful in pondering the nature of this art.
Georg Wilhelm Friedrich Hegel made one of my favorite remarks about Shakespeare when he observed that his great characters are “free artists of themselves.” That is to say that Shakespeare’s most memorable men and women—beginning with characters like Berowne in Love’s Labour’s Lost and the titular tyrant of Richard III, and climaxing with amazingly vital figures like Hamlet, Macbeth, Falstaff (Henry IV), and Iago (Othello)—are improvisational poets, clowns, raconteurs, and philosophers of amazing brilliance, performing in the dramas of their own lives. They strive with the full force of their souls to author, analyze, revise, and direct this drama before our very eyes, to alter it, control it, or simulate and reproduce it (like Hamlet with his Mousetrap). The entire plot of Othello is essentially an elaborate deception, a play within a play devised and improvised by Iago, with Othello, Desdemona, and Cassio ensnared as his unwitting actors.
The second critical notion I want to reference for you is one from Harold Bloom, who quotes that wonderful line of Hegel’s in his book Shakespeare: The Invention of the Human. Bloom locates Shakespeare’s ultimate originality of characterization in a phenomenon he calls “self-overhearing” and discusses throughout the book, the phenomenon by which a character (such as Hamlet) observes himself speaking, or thinking, and is changed or enlightened by this process of self-expression and self-definition.
Bloom’s idea is closely related to Hegel’s; Shakespeare’s characters are aware of themselves as dramatic personalities, players in a drama they don’t fully comprehend but which they are trying to revise, enacted on a stage as large as the whole history of the human race. The soliloquy is Shakespeare’s primary tool in portraying the art of self-overhearing—an art, Bloom argues (and I agree), that is not confined to Shakespeare but that every one of us practices, alone or in the presence of others, every day.
What makes Shakespeare’s characters so very different from us—even as they feel things we feel and help us to understand ourselves more deeply—is that the soliloquy allows them to express their inmost thoughts and conflicts through language, whereas most of us, when we are alone, think through difficult matters silently, in a verbally vague and fragmentary way. This, too, marks them as “artists.” Diction, word choice, and poetic language are the wellspring from which Shakespearean character arises.
Listening to these characters think about and try to define their own lives and their own natures, you discover zestful and visionary creative minds in action. Shakespeare was not just a great artist, but a creator of great artists, with desires, values, ambitions, and imaginative sensibilities all their own.