Who does Caliban tell about Prospero having him as a slave?

In act 3, scene 2, Caliban tells Stephano about having had the island stolen from him by Prospero. Because Stephano's friend, Trinculo, happens to be there, he also hears the story. Trinculo does not believe Caliban and calls him a liar.
According to Caliban's telling of the story, Prospero, a tyrannous and wicked magician, used deceitful magic spells to take the island from him and make him a slave. Caliban tells this story in order to try to entice Stephano into kiling Prospero and taking Miranda as his bride. Caliban says,

As I told thee before, I am subject to a tyrant, a sorcerer that by his cunning hath cheated me of the island.

However, it is in a conversation with Prospero early in the play, in act 1, scene 2, that Caliban tells a fuller story of his enslavement. He says that Prospero wooed him with kind words and then betrayed him by taking his island and enslaving him, making Caliban his subject. Prospero objects and says he only turned on Caliban when Caliban tried to rape Miranda. Caliban says:

This island’s mine, by Sycorax my mother,
Which thou takest from me. When thou camest first,
Thou strok’st me and made much of me . . .
And then I loved thee
And showed thee all the qualities o' th' isle,
The fresh springs, brine pits, barren place and fertile.
Cursed be I that did so! All the charms
Of Sycorax, toads, beetles, bats, light on you!
For I am all the subjects that you have,
Which first was mine own king.