Rand muses on this question before learning to Travel
In any case, traveling required that you know your starting point very well. It seemed more logical to him that you should have to know where you were heading well, but Asmodean seemed to think that was asking why air was not water. There was a great deal Asmodean took for granted.
The Fires of Heaven, chapter 6, Gateways
Asmodean's response isn't terribly satisfying. Perhaps it's very obvious why once you understand the underlying principles, perhaps Asmodean has had a long time to become accustomed to the answer, or perhaps he was never curious about it.
However, Rand does give us some explanations of how Traveling works after he figures it out:
He had no way to describe what he did except as making a fold in the Pattern, a hole through it.
Lord of Chaos, chapter 3, A Woman's Eyes
He took the end of her shawl in both hands. "The Pattern," he said. "Caemlyn," one finger on his left hand tented the wool, "and Cairhien." A finger on the other hand made a tent, and he brought the two tents together. "I bend the Pattern and bore a hole from one to the other. I don't know what I bore through, but there's no space between one end of the hole and the other."
LoC, chapter 27, Gifts
Again, no real explanation for knowing the start is offered. It seems to be a magical fact that the hole can only be bored from the starting side.
Traveling with Saidar works quite differently. Egwene makes a good guess:
She thought the way would be to create-a similarity seemed the only way to describe it-a similarity between the between the real world and its reflection in the World of dreams. That should make a place where it was possible to simply step from one to the other.
LoC, chapter 27, Gifts
This idea is close enough to the reality that she makes a Gateway with only minimal explanation from Moghedien
"You make the two places in the Pattern identical."
LoC, chapter 37, When Battle Begins
This likewise doesn't really explain why one needs knowledge of one side in particular.
Out of universe, I've always felt it was a way to have the characters learn the lost powers of ages past more gradually. Skimming is introduced significantly earlier, but nearly disappears once Traveling is rediscovered (Compare the 37 instances of Traveling to the 8 of Skimming listed on TarValon.Net).
The fact that there was once a faster way to go from place to place (that their enemies are actively using) and how nice it would be if they could relearn it is something characters think about quite often, even as they use the Ways, Portal stones, and Skimming to improve on mundane travel.