TL;DR: It's never made explicit in the text itself but there are some options for who it is likely to be. I will list them roughly in order of most to least likely though Tywin Lannister is the most likely candidate.
Ned had to make common cause with the Lannister's after Robert's Rebellion due to Tywin funding the Seven Kingdoms and Robert having married Cersei. He doesn't appear to have gotten over some of their actions though.
Tywin Lannister
He makes it clear in these passages that he distrusts Tywin and has no care for him even going as far as to be prepared to insult him.
Ned would sooner entrust a child to a pit viper than to Lord Tywin, but he left his doubts unspoken. Some old wounds never truly heal, and bleed again at the slightest word. "The wife has lost the husband," he said carefully. "Perhaps the mother feared to lose the son. The boy is very young."
...
"A generous offer, my friend," the king said, "but too late. Lord Tywin has already given his consent. Fostering the boy elsewhere would be a grievous affront to him."
"I have more concern for my nephew's welfare than I do for Lannister pride," Ned declared.
A Game of Thrones, Eddard I
He also thinks of Tywin as a murderer of innocents and appears to despise him still for giving the orders to kill the children.
Ned did not feign surprise; Robert's hatred of the Targaryens was a madness in him. He remembered the angry words they had exchanged when Tywin Lannister had presented Robert with the corpses of Rhaegar's wife and children as a token of fealty. Ned had named that murder; Robert called it war. When he had protested that the young prince and princess were no more than babes, his new-made king had replied, "I see no babes. Only dragonspawn." Not even Jon Arryn had been able to calm that storm. Eddard Stark had ridden out that very day in a cold rage, to fight the last battles of the war alone in the south. It had taken another death to reconcile them; Lyanna's death, and the grief they had shared over her passing.
This time, Ned resolved to keep his temper. "Your Grace, the girl is scarcely more than a child. You are no Tywin Lannister, to slaughter innocents." It was said that Rhaegar's little girl had cried as they dragged her from beneath her bed to face the swords. The boy had been no more than a babe in arms, yet Lord Tywin's soldiers had torn him from his mother's breast and dashed his head against a wall.
A Game of Thrones, Eddard II
He also thinks of the man as dishonourable for when he betrayed the Mad King during the Sack of King's Landing.
"Not our men," Ned said patiently. "Lannister men. The lion of Lannister flew over the ramparts, not the crowned stag. And they had taken the city by treachery."
The war had raged for close to a year. Lords great and small had flocked to Robert's banners; others had remained loyal to Targaryen. The mighty Lannisters of Casterly Rock, the Wardens of the West, *had remained aloof from the struggle**, ignoring calls to arms from both rebels and royalists. Aerys Targaryen must have thought that his gods had answered his prayers when Lord Tywin Lannister appeared before the gates of King's Landing with an army twelve thousand strong, professing loyalty. So the mad king had ordered his last mad act. He had opened his city to the lions at the gate.
A Game of Thrones, Eddard II
Jaime Lannister
Like Tywin he doesn't appear to trust Jaime very much hoping that Robert won't name him Warden to the East because that would mean he would eventually hold the Est and West.
"Kingslayer," Ned said. The rumors were true, then. He rode on dangerous ground now, he knew. "An able and courageous man, no doubt," he said carefully, "but his father is Warden of the West, Robert. In time Ser Jaime will succeed to that honor. No one man should hold both East and West." He left unsaid his real concern; that the appointment would put half the armies of the realm into the hands of Lannisters.
A Game of Thrones, Eddard II
Again he reiterates that he doesn't trust Jaime and even spells it out clearly for Robert. He even thinks he is dishonourable for sitting himself atop the Iron Throne. He also appears to dislike Jaime's arrogance and cockiness.
"Can you trust Jaime Lannister?"
"He is my wife's twin, a Sworn Brother of the Kingsguard, his life and fortune and honor all bound to mine."
His sword helped taint the throne you sit on, Ned thought, but he did not permit the words to pass his lips. "He swore a vow to protect his king's life with his own. Then he opened that king's throat with a sword."
"Seven hells, someone had to kill Aerys!" Robert said, reining his mount to a sudden halt beside an ancient barrow. "If Jaime hadn't done it, it would have been left for you or me."
...
"I cannot answer for the gods, Your Grace … only for what I found when I rode into the throne room that day," Ned said. "Aerys was dead on the floor, drowned in his own blood. His dragon skulls stared down from the walls. Lannister's men were everywhere. Jaime wore the white cloak of the Kingsguard over his golden armor. I can see him still. Even his sword was gilded. He was seated on the Iron Throne, high above his knights, wearing a helm fashioned in the shape of a lion's head. How he glittered!"
"This is well known," the king complained.
"I was still mounted. I rode the length of the hall in silence, between the long rows of dragon skulls. It felt as though they were watching me, somehow. I stopped in front of the throne, looking up at him. His golden sword was across his legs, its edge red with a king's blood. My men were filling the room behind me. Lannister's men drew back. I never said a word. I looked at him seated there on the throne, and I waited. At last Jaime laughed and got up. He took off his helm, and he said to me, 'Have no fear, Stark. I was only keeping it warm for our friend Robert. It's not a very comfortable seat, I'm afraid.'"
The king threw back his head and roared. His laughter startled a flight of crows from the tall brown grass. They took to the air in a wild beating of wings. "You think I should mistrust Lannister because he sat on my throne for a few moments?" He shook with laughter again. "Jaime was all of seventeen, Ned. Scarce more than a boy."
"Boy or man, he had no right to that throne."
ibid
Gregor Clegane
It can be argued Ned has never really made common cause with Gregor but he has rode alongside him and been forced to put up with him due to their common allegiance with the Lannisters. Ned appeared to dislike and distrust Gregor and even put some belief into the rumours over the knight and that he was the one to kill Elia and the babes.
Unlike his brother, Ser Gregor did not live at court. He was a solitary man who seldom left his own lands, but for wars and tourneys. He had been with Lord Tywin when King's Landing fell, a new-made knight of seventeen years, even then distinguished by his size and his implacable ferocity. Some said it had been Gregor who'd dashed the skull of the infant prince Aegon Targaryen against a wall, and whispered that afterward he had raped the mother, the Dornish princess Elia, before putting her to the sword. These things were not said in Gregor's hearing.
Ned Stark could not recall ever speaking to the man, though Gregor had ridden with them during Balon Greyjoy's rebellion, one knight among thousands. He watched him with disquiet. Ned seldom put much stock in gossip, but the things said of Ser Gregor were more than ominous. He was soon to be married for the third time, and one heard dark whisperings about the deaths of his first two wives. It was said that his keep was a grim place where servants disappeared unaccountably and even the dogs were afraid to enter the hall. And there had been a sister who had died young under queer circumstances, and the fire that had disfigured his brother, and the hunting accident that had killed their father. Gregor had inherited the keep, the gold, and the family estates. His younger brother Sandor had left the same day to take service with the Lannisters as a sworn sword, and it was said that he had never returned, not even to visit.
A Game of Thrones, Eddard VII
Roose Bolton
I am mentioning Roose here because he's mentioned in the question and there's a slim chance it his him but I highly doubt it. In fact I'm pretty sure it isn't supposed to be him.
The Stark's and the Bolton's have been in conflict with each other for about as long as there have been Stark's and Bolton's. So you'd assume this would be a likely candidate and the two have had to kiss and make up for Robert's Rebellion, the Greyjoy Rebellion and in most day to day running's of the North.
However, the only mention of Roose, or any Bolton for that matter, in eddard's chapters is the following. It's not a positive thought of Ned's though.
"Mercy is never a mistake, Lord Renly," Ned replied. "On the Trident, Ser Barristan here cut down a dozen good men, Robert's friends and mine. When they brought him to us, grievously wounded and near death, Roose Bolton urged us to cut his throat, but your brother said, 'I will not kill a man for loyalty, nor for fighting well,' and sent his own maester to tend Ser Barristan's wounds." He gave the king a long cool look. "Would that man were here today."
A Game of Thrones, Eddard VIII
Rhaegar Targaryen is mentioned in the question so I'll address him here but I don't think Ned despised him. In fact most of his recollections of Rhaegar actually appear to be quite positive, for example...
There was no answer Ned Stark could give to that but a frown. For the first time in years, he found himself remembering Rhaegar Targaryen. He wondered if Rhaegar had frequented brothels; somehow he thought not.
A Game of Thrones, Eddard IX
In fact from seeing Rhaegar give Lyanna the queen of beauty's laurel instead of his own wife and knowing about their relationship. Even going as far as to
look after Jon and hide him away.
It's likely that Ned had some respect for the man and probably didn't despise him just out of respect for Lyanna.
Robert had been jesting with Jon and old Lord Hunter as the prince circled the field after unhorsing Ser Barristan in the final tilt to claim the champion's crown. Ned remembered the moment when all the smiles died, when Prince Rhaegar Targaryen urged his horse past his own wife, the Dornish princess Elia Martell, to lay the queen of beauty's laurel in Lyanna's lap. He could see it still: a crown of winter roses, blue as frost.
A Game of Thrones, Eddard XV
