I have partially read The Giver, but I still don't get why The Giver gets to keep the memories of people.
You can find the answer directly in the book. The Giver explains this in chapter 14.
“Why?” Jonas asked him after he had received a torturous memory […] “Why do you and I have to hold these memories?”
“It gives us wisdom,” The Giver replied. “Without wisdom I could not fulfill my function of advising the Comittee of Elders when they call upon me.”
“But what wisdom do you get from hunger?” Jonas groaned. […]
“Some years ago,” The Giver told him, “before your birth, a lot of citizens petitioned the Committee of Elders. They wanted to increase the rate of births. They wanted each Birthmother to be assigned four births instead of three, so that the population would increase and there would be more Laborers available.”
Jonas nodded, listening. “That makes sense.”
“The idea was that certain family units could accommodate an additional child.”
“The Comittee of Elders sought my advice,” The Giver said. “It made sense to them, too, but it was a new idea, and they came to me for wisdom.”
“And you used your memories?”
The Giver said yes. “And the strongest memory that came was hunger. It came from many generations back. Centuries back. The population had gotten so big that hunger was everywhere. Excruciating hunger and starvation. It was followed by warfare.”
Warfare? It was a concept Jonas did not know. But hunger was familiar to hum now. Unconsciously he rubbed his own abdomen, recalling the pain of its unfulfilled needs. “So you described that to them?”
“They don't want to hear about pain. They just seek the advice. I simply advised them against increasing the population.”
“[…] I knew that there had been times in the past – terrible times – when people has destroyed others in haste, in fear, and had brought about their own destruction.”
Jonas realized something. “That means,” he said slowly, “that you have memories of destruction. And you have to give them to me, too, because I have to get the wisdom.”
The Giver nodded.
“But it will hurt,” Jonas said. It wasn't a question.
“It will hurt terribly,” The Giver agreed.
“But why can't everyone have the memories? I think it would seem a little easier if the memories were shared. You and I wouldn't have to bear so much by ourselves, if everybody took a part.”
The Giver sighed. “You're right,” he said. “But then everyone would be burdened and pained. They don't want that. And that's the real reason the Receiver is so vital to them, and so honored. They selected me – and you – to lift that burden from themselves.”
“The decision was made long before my time or yours,” The Giver said, “and before the previous Receiver, and –” He waited.
The community has decided, generations before this story, that they don't want everyone to share all the pain that comes with the memories of an imperfect past. They moved all that responsibility to a single individual, the Giver, who is honored but also isolated form the rest of the society. Since that decision has happened much before the start of the story, the above quotes are all we can know about the reasons.
During this story,
the Giver and the Receiver has decided to finally change this system, and distribute the memories among all the population to share once again. They did this not only for their own well-being (which has caused serious pain to them, as shown in chapters 13 and 14), but because they believed it would be better for the community in the long run, and they could help the transition.
Spoiler (from the very end of the book so if you don't want to find out what happens then don't read this - I've just put it here for people who give up reading at some point during the novel).
The rest of the society has chosen to forget everything bad (and good) about life in order to have everything equal. Think communism taken to the very extreme. There are no colours so people can compare things (who has the reddest apple, who has the best clothes) and so that avoids jealousy. No sexual feelings so people aren't driven by them, couples are picked for compatibility and not love etc. However, the Giver has been chosen for the capacity to hold memories for the entire community about what life was like before "Sameness". In that way, one person would be able to give advice about how the community could stay safe (avoiding war and that sort of thing). I don't have the book in front of me but I remember the Giver saying that the council wanted to do something and he advised against it since the decision might lead to war.