How do we know that the reaper's song had a lasting effect on the poet ?

The easiest way to know that the solitary reaper's song stays with the speaker of the poem is to look at the last lines of "The Solitary Reaper":

I listened, motionless and still; 
And, as I mounted up the hill, 
The music in my heart I bore, 
Long after it was heard no more. (29-32)

From these lines, we can tell that the speaker of the poem is continuing to think about the solitary reaper's song even after he walks away. The connotation here is that the song has deeply affected the speaker (who is a stand-in for the poet's own voice and thoughts and feelings), and that the song continues to move him for a long time afterwards. The song is significant to the speaker because it conjures up an infinite array of transcendent meaning, and so the suggestion is that, through the singing solitary reaper, the speaker has encountered a moment of spiritual/existential significance. The idea that one could encounter ultimate meaning in everyday life (especially everyday life that involved a setting of natural beauty) was a key Romantic ideal.
https://www.poetryfoundation.org/poems/45554/the-solitary-reaper