It's mentioned that Jon Arryn was like a father to Ned and Robert, as he brought them up.
By why did they grow up with him?
Science Fiction & Fantasy Stack Exchange is a question and answer site for science fiction and fantasy enthusiasts. It only takes a minute to sign up.Sign up to join this community
This is the Westerosi version of the medieval European practice of fostering:
Later in their childhood, between the ages of 8 and 10, boys – and occasionally girls, too – of knightly and noble status would be sent away to live with another noble family for a few years in a custom called fostering. Sometimes the foster family would be related to the birth parents; William Marshal fostered with his cousin William of Tancarville. More often than not, though, foster parents were chosen for political rather than familial connections. The royal household was an especially prestigious place to foster; some royal households had upwards of 100 young men and women in their care.
Fostering helped young nobles make connections with other families while training them in the skills they would need as adult members of the aristocracy. Spending time with young people of a similar age and social class would – or so their parents hoped – strengthen the political bonds among the next generation of the nobility, and maybe even enable them to find a potential spouse.
As Lord of the Vale, Jon Arryn was one of the most powerful noblemen in the Seven Kingdoms. On a personal level, he was well known for his wisdom and sense of honour. Therefore, he was a logical choice to foster the sons of similarly powerful families, such as the Starks and Baratheons.
They were sent by their parents to be "fostered" by Lord Arryn. This is a common practice among the noble families of Westeros, as it creates better ties among close allies (including better potential for marriages) and ensures the loyalty of not-so-close allies (the children are basically hostages in that case, an example is Theon Greyjoy).