Who is "The Raven" addressed to?

If "The Raven" is addressed specifically to anyone, it would be to people such as Poe himself whose mental world inhabits a region that's a mixture of fact and dream, of the world and a fantasy that partakes of both heaven and hell.
The speaker is a man who, probably like Poe himself and perhaps most other writers and artists in general, lives primarily in the night. While studying, pondering at midnight instead of sleeping, the raven intrudes upon him from the realm of dreams. He is already reveling in "fantastic terrors," and moreover seems in a trance-like state, repeating the strangely musical words over and over again. The raven initially offers hope. The speaker has been utterly alone in his murmuring form of insomnia; the visitor appears at the door and speaks to him, but only, of course, with the now-famous answer "nevermore."
Those to whom Poe addresses the poem are the ones who themselves feel they can answer nothing, beyond that cryptic word, to their own existential questions. The fate of the "lost Lenore," and a possible reunion with her is unknown. The fate of the speaker's own psyche is unknown as well, except for his conviction that it will be perpetually in darkness:

And my soul from out that shadow that lies floating on the floor,
Shall be lifted--Nevermore!

As always in Poe, the mere sound of the words has an importance that rivals their meaning. The reader is caught in a web of music, a jingling sound that in the hands of a lesser poet might descend into doggerel. So finally I would suggest that the verses are directed to the reader susceptible to this kind of extra-linguistic feature of Poe, to a kind of musical mesmerism that seeks to soothe and terrify in the same moment.

The poem "The Raven" is itself not addressed to anyone in particular; it does not contain any instances of apostrophe, nor does it address the raven directly. Rather, its narrator seems to be telling an anecdote to an unknown audience in the fashion of a "fireside story" or supernatural tale. The narrator tells of "once upon a midnight dreary," beginning his tale in traditional storytelling format and setting it in "the bleak December," which lends to the gloomy atmosphere.
The narrator does, however, relate how he addressed the raven--when it was an unknown presence "tapping at [his] chamber door," the narrator addresses him as "Sir...or Madam," but found "darkness there and nothing more." The narrator then brings himself to question whether the presence might be Lenore, his lost love, until eventually "there stepped a stately Raven" into his sight. At this point, the narrator addresses the Raven directly: "Tell me what they lordly name is," receiving the much-quoted answer, "Nevermore."