What does the term “bah! humbug!” mean?

Scrooge seems to use the term to describe anything that he thinks of as nonsense.  It's his way of expressing his disdain for whatever subject is being discussed or whatever event is at hand.  When Scrooge's nephew, Fred, comes to wish him a Merry Christmas, Scrooge replies with "Bah! [...] Humbug!"  Thus, we begin to understand that he thinks celebrating Christmas is ridiculous.  
Next, Fred tells Scrooge that, by his own logic, he has no reason to be gloomy because he's "rich enough."  That elicits another "Bah! [...] Humbug."  This could indicate that, for Scrooge, it isn't possible to be rich enough or too rich.  He just wants to acquire and gain and gain acquire more and more.
Later, in his own home, Scrooge hears the bells all through the house ringing and ringing, a sound that is frightening because he believes himself to be all alone.  But when he hears the heavy clanking of dragging chains come up from the cellar toward his own door, he says, "It's humbug still! [...] I won't believe it."  He will soon tell Marley that he doesn't believe in the ghost because one's senses can cheat, making a person see something not really there.  He thinks the ghost is nonsense, at least at first, and so he says humbug.  Scrooge even tries to say it again at the end of this first stave, but he "stopped at the first syllable": no longer quite able to disbelieve what he has seen tonight, he can no longer refer to it as nonsense.

"Bah!" is an interjection expressing contempt. The word "humbug," as well as meaning a kind of boiled sweet, also refers to false, deceptive talk or behavior. That's what Scrooge thinks about people getting all excited about the Christmas season. To him, it's just a complete waste of time, time that could be much better spent in making money. Scrooge cannot understand why everyone else doesn't see through what he regards as the sheer phoniness of the Christmas holidays.
In stave 1, his nephew Fred comes round to wish him the compliments of the season and invite him over for Christmas dinner. But Scrooge, being Scrooge, isn't having any of it. That's when he lets fly with the insult that has become almost a catchphrase for anyone not entering into the Christmas spirit.