In The Signalman by Charles Dickens, what scares the narrator when he looks down the mouth of the train tunnel?

For the first time, the narrator sees the sight that the signal-man had described to him earlier:

Close at the mouth of the tunnel, I saw the appearance of a man, with his left sleeve across his eyes, passionately waving his right arm.

This seizes the narrator with a "nameless horror" because he believes he is seeing the ghost whose appearances have tormented the signal-man. This quickly passes when he realizes he is seeing one of a group of men investigating an accident scene: The signal-man has been killed by a train.
Dickens has laid the groundwork for this terror by his eerie, sympathetic, and lonely portrayal of the signal-man. Reviewing this background helps us understand why the narrator felt fear.
The signal-man works in a bleak and desolate place, and believes he has been visited by a ghostly apparition on repeated occasions. The signal-man believes the ghost is attempting to announce impending disaster or death. Twice, the ghost's appearance was followed by death on the railway.
The narrator is moved by the signal-man's devotion to duty, sincerity, and feelings of helplessness to prevent future deaths. He is also affected by the creepiness of the signal-man's stories. He repeatedly shakes off his own fear and attempts to find rational explanations for the signal-man's experiences:

Resisting the slow touch of a frozen finger tracing out my spine, I showed him how that this figure must be a deception of his sense of sight. . .
A disagreeable shudder crept over me, but I did my best against it. It was not to be denied, I rejoined, that this was a remarkable coincidence, calculated deeply to impress his mind. But it was unquestionable that remarkable coincidences did continually occur.

After the two men peer into the tunnel together and see nothing, the narrator seems to shake off his doubts entirely. His concern is to weigh public safety against his loyalty to the signal-man: It seems the signal-man is suffering from psychological problems that could lead to a mistake on the job, which could put train passengers at risk. He doesn't want to report the signal-man to his superiors and get him into trouble, so he resolves to help him meet with a doctor.
This, then, is the context in which the narrator finally sees the apparition, or at least momentarily believes he is seeing the apparition. Suddenly he sees the very thing he had convinced himself doesn't exist.
As it turns out, the man he sees isn't the ghost, but the circumstances lead the narrator (and reader) to believe that eerie, supernatural forces have indeed been at work.