Card actually explained the real reason eventually, and I bumped into it when re-reading Shadow of the Giant.
In conversation between Peter Wiggin and Mazer Rakham, Mazer finally owns up to the real reason:
Rackham laughed. "Peter, Graff was so right about you."
"When he rejected you for Battle School."
"Because I was too aggressive," said Peter wryly. "And look what he actually accepted."
"Peter," said Rackham. "Think about what you just said."
Peter thought about it. "You mean about juggling."
"I mean about why you were rejected for Battle School."
Peter immediately felt stupid. His parents had been told that he was rejected because he was too aggressive—dangerously so. And he had wormed that information out of them at a very young age. Ever since then, it had been a burden he carried around inside—the judgment that he was dangerous. Sometimes it had made him bold; more often, it had made him not trust his own judgment, his own moral framework. Am I doing this because it's right? Am I doing this because it will really be to my benefit? Or only because I'm aggressive and can't stand to sit back and wait? He had forced himself to be more patient, more subtle than his first impulse. Time after time he had held back. It was because of this that he had used Valentine and now Petra to write the more dangerous, demagogic essays—he didn't want any kind of textual analysis to point to him as the author. It was why he had held back from any kind of serious arm-twisting with nations that kept playing with him about joining the FPE—he couldn't afford to have anyone perceive him as coercive.
And all this time, that assessment of him was a lie.
"I'm not too aggressive."
"It's impossible to be too aggressive for Battle School," said Rackham. "Reckless—now, that would be dangerous. But nobody has ever called you reckless...
"No, Graff looked at your tests and watched what the monitor showed us, and then he talked to me and showed me, and we realized: You weren't what we wanted as commander of the army, because people don't love you. Sorry, but it's true. You're not warm. You don't inspire devotion. You would have been a good commander under someone like Ender. But you could never have held the whole thing together the way he did."
"I'm doing fine now, thanks."
"You're not commanding soldiers. Peter, do Bean or Suri love you? Would they die for you? Or do they serve you because they believe in your cause?"
"They think the world united under me as Hegemon would be better than the world united under anyone else, or not united at all."
"A simple calculation."
"A calculation based on trust that I've damn well earned."
"But not personal devotion," said Rackham. "Even Valentine—she was never devoted to you, and she knew you better than anyone."
Rackham nodded. "But one thing was certain. Your off-the-charts aggressiveness, your passion to control events, we knew that you would place yourself in the center of everything."
It was Peter's turn to laugh. "So you left me home from Battle School so I would be what I am now."
"As I said, you weren't suited for military life. You don't take orders very well. People aren't devoted to you, and you aren't devoted to anyone else."
"I might be, if I found somebody I respected enough."
"The only person you ever respected that much is on a colony ship right now and you'll never see him again."
"I could never have followed Ender."
"No, you never could. But he's the only person you respected enough. The trouble was, he was your younger brother. You couldn't have lived with the shame."