Because Bran had no connection whatsoever with Ghost or any training on forging a connection with another beast.
Why show can't answer this
By show-only perspective, you won't get any answers because Show does not discuss the skin-changing lore in detail, does not provide a historical overview and does not give examples.
That is why we have to rely on books for this:
Books Perspective
Bran could not call Greywind or Shaggydog away from Tyrion
The Stark Children show the gift, at least Robb, Rickon, Arya, Jon and Bran do. When Tyrion Lannister was set upon by Stark wolves, Robb, Rickon and Bran all had to call their beasts off. No one among them could call other's beast off. As we saw in AGOT in Bran's POV:
“No!” Bran shouted from the high seat as Lannister’s men reached for
their steel. “Summer, here. Summer, to me!”
The direwolf heard the voice, glanced at Bran, and again at Lannister.
He crept backward, away from the little man, and settled down below
Bran’s dangling feet.
Robb had been holding his breath. He let it out with a sigh and
called, “Grey Wind.” His direwolf moved to him, swift and silent.
Now there was only Shaggydog, rumbling at the small man, his eyes
burning like green fire. “Rickon, call him,” Bran shouted to his baby
brother, and Rickon remembered himself and screamed, “Home, Shaggy,
home now.”
As you can see, neither of them shared a bond with each others wolf and thus could not command any wolf other than their own to leave the Lannister alone. So if Bran could not control Shaggy dog or Greywind, we can assume he could never have controlled Ghost either.
Distance between Ghost and Bran
We know that distance between the canine-beast and the warg does not matter if the warg has already claimed the beast or has a bond with it. They have not shown this in the show, but in books during the ranging beyond the wall with Qhorin Halfhand, Jon warged into Ghost in his sleep who was miles away and saw the Army of Wildlings. Also, Arya Stark could connect with Nymeria's mind while being in Braavos but only if she was asleep.
But Jon was able to do that because he had a bond with ghost. Skinchangers can take as many beasts as they want (e.g. Varamyr Sixskins) but they have to fight the animal for control and form that bond. So in theory, Bran could warg into Ghost if he knew how to form the bond with him.
Bran was untrained by the time he saw Jon and when urged by Jojen, he warged using the only bond he had; That's with Summer. He had no bond with Ghost so I doubt he could even feel him, let alone slip into him.