To expand upon @Juan Manuel's answer a bit:
Walter originally built a device that could view other parallel universes, based on the Fringe universe's version of the many worlds theory. He used that device to watch alternate versions of himself try to solve the problem of Peter's fatal disease, something that he was unable to do in time. Walter found many such universes, none of them sucessfully curing Peter. He was focused in on one particular world where the alternate-him was close to solving the problem when things went sideways.
He just happened to see Walternate's experiment succeed, at the same moment that Observer September distracted Walternate, causing him to miss the result. Walter then decided that he needed to actually go to that other universe and save Peter. (Fully intending, I believe, to return Peter when he was done.) That's when Walter built the actual bridge device to travel to that universe and retrieve Peter, cure him, and return him.
When he finally went through with his plan, the end result was that Peter stayed in our world, and now there was an imbalance between the two worlds. That tied those two specific universes together in a way that other universes were not. Walter closed the bridge, but it was still "there" in some sense that the two universes were connected and "moving together".
In theory, Walter's bridge device could take someone to any other universe, assuming they were able to "find" it and calibrate the machine correctly. It's simply that our universe and the prime-universe were the easiest to bridge (having been done once), and there was little incentive for any anyone from either universe to find a third one.