What does Baldwin mean by "black people, mainly, look down or look up but do not look at each other, not at you, and white people, mainly, look away"?

What Baldwin is referring to is the fact that racism separates black people, not just from whites, but from each other. The indignities of racism have instilled a certain mindset in African Americans that makes them seek a kind of artificial dignity in class distinctions. Essentially, they internalize society's racism and then use it to construct barriers between themselves and other African Americans on the basis of class. So, as well as constantly having to "look up" at white people, lower-class African Americans also have to do the same to their alleged social superiors of the same race.
Among other things, this means that black people are divided from each other when they should be united. Instead of coming together to fight racism and oppression, they're too busy trying to create a distinct social hierarchy that keeps a whole segment of black society down, oppressed by both white racism and black middle-class snobbery. In the meantime, white society can just look away, ignoring what goes on among African Americans, safe in the knowledge that, in this unequal society, and with black America divided among itself, white society remains firmly in control.

Baldwin, in my view, is stating that many or most black people are unable actively to come to grips with the general conditions in which African Americans have been placed by white society. At the time he wrote The Fire Next Time,the civil rights movement was only in its infancy. It still appeared to many that there were no obvious solutions to racial oppression, and in Baldwin's opinion, African Americans, in "not looking at each other," were still unable to evaluate their situation in objective terms and still felt largely powerless to effect change.
In saying "white people look away," Baldwin is indicating that most whites simply do not wish to acknowledge the fact of racial oppression and, still less, to take any action to correct it. He also means that too many whites (up to that time and, tragically, still today) have considered African Americans to be invisible or have regarded them as somehow not truly a part of America. Though much progress has been made in the fifty-five years since The Fire Next Time was published, many of the points Baldwin makes are unfortunately still valid today.

Baldwin is addressing the way in which race has created a hierarchy whose strictures are obeyed by both whites and blacks. 
Black people "look up" at those within the race who are of a higher economic class and/or whose skin is of a lighter hue. They "look down" on those who are poorer or darker. Black men might be inclined to "look down" on black women; heterosexuals may also "look down" on homosexuals.
Racism, sexism, homophobia, and classism, which often coincide, have dissuaded efforts at black unity and even the basic acknowledgement that another black person's life, however degraded or different, is still important and invaluable. If members of the community were able "to look at each other," as fellow individuals, this understanding would be obvious. The "not at you" is a reference to Baldwin's nephew, to whom the letter in The Fire Next Time is addressed.
Whites, he argues, "look away," unable to see the presence of black people at all. To look away from someone is to refuse to acknowledge their existence. In his impassioned argument, Baldwin spares no white American—neither conservatives nor liberals—from his accusation that they do not think that black lives matter as much as their own; he also says that their concerns over rioting and unrest at the time were not in response to conditions in ghettos, but instead a reaction to possible intrusion onto their property. When one's first response, in face of another's pain, is to talk about oneself, there is not only no concern for the other, there is no sense that he is there at all.