Was Mafatu's conflict with Moana or himself the greater battle in Call it Courage by Armstrong Sperry?

It's really a combination of both. Mafatu must overcome his own deep-seated fears—fears of the ocean, fears of being a failure, and fears of being regarded as a coward by his own people—if he's to overcome Moana, the great sea god.
To some extent, Mafatu's fear of Moana is a projection of his many other fears; Moana is the scariest thing that Mafatu's ever encountered, not least because of his extraordinary power. As Mafatu comes to realize over the course of his adventures, you need to fight fire with fire; he must respond to Moana's power with power of his own—the power of courage and endurance. But before he can do that, he must somehow resolve the many conflicts that rage within his soul. And one could argue that Mafatu's inner conflicts are so much more difficult to resolve than his external conflict with Moana, even though they are all intimately related.

There really is no correct answer to this question. You have to decide for yourself which of these conflicts was a greater battle for Mafatu in Call It Courage.
On the one hand, we can argue the “man vs. nature” battle in Call It Courage is the greater battle. Mafatu has to struggle in a very important way against Moana. When he is little, Moana kills Mafatu's mother and almost takes him. When Mafatu runs away from Hikueru, Moana sends a terrible storm and comes very close to killing him. When Mafatu flees the cannibals and tries to return to Hikueru, Moana once again comes very close to killing him by becalming him and leaving him to potentially die of thirst and hunger. This means Moana almost kills him on three separate occasions. This is surely a great battle.
On the other hand, we can also claim “man vs. self” is the greater battle here. Mafatu does not have to battle himself in order to survive. He could easily survive by staying on Hikueru and continuing to live as he has. He does have to battle himself so he can live a rewarding life. He has to fight against his fears. If he does not, he will never be recognized as a man on Hikueru. This is not a literal matter of life or death, but it is something that will determine Mafatu's quality of life.
In order to answer this question, then, you have to decide what the greater battle is. Is the harder battle one where you fight outside forces to save your life, or is it the one where you fight yourself and try to conquer your own fears and weaknesses in order to improve yourself?