Are you thinking of Pandora's Star by Peter F Hamilton, and its sequel Judas Unchained?
It is a much larger story than you've indicated. But it has the basic plot elements you cited:
Human driven plot elements that match:
- Astronauts go to Mars via conventional spaceship technology
- Inventors of new wormhole technology also make an instantaneous trip to Mars to "surprise" the astronauts (as their announcement of their invention)
- That wormhole technology leads to a vast human empire on many worlds, using commuter trains and cargo trains that pass through the pre-established human wormholes on the human worlds
Silfen (alien) driven subplot elements that match:
- An alien race (the Silfen) has a system of "mystical paths" that lead from world to world, but selected humans can use these paths too; these paths are later revealed to be very advanced dynamic wormhole-like technology
- One of the wormhole inventors (Ozzie Isaacs) does indeed follow the Silfen's paths from world to world, unable to get back to where he started
- Finally Ozzie winds up on a large "flat world" that has an ocean of water dropping over the edge, and as the first novel's cliffhanger his ship drops over the edge
These two novels lead into Hamilton's Void series novels set in the same universe and extended timeline (which are now expanding beyond the original Void trilogy). Its a pretty wonderful and complex mostly "hard SciFi" series of books that explore many themes and consequences of a commonwealth of planets, instantaneous travel, instantaneous communications, longevity, future terrorism, alien invasion, multi-world corporations, multi-generational family power-bases, future criminal investigation techniques, future criminal punishment, sentient artificial intelligence, etc.
Some readers dislike the Void series because they contain what appears to be a fantasy aspect. However as Arthur C Clarke stated "Any sufficiently advanced technology is indistinguishable from magic". And indeed the aspect turns out not to be fantasy.