They use the stars
Maester Luwin sighed. "I can teach you history, healing, herblore. I can teach you the speech of ravens, and how to build a castle, and the way a sailor steers his ship by the stars. I can teach you to measure the days and mark the seasons, and at the Citadel in Oldtown they can teach you a thousand things more. But, Bran, no man can teach you magic."
A Game of Thrones, Bran VI
Davos is also very familiar with the stars and refers to them as smuggler's stars suggesting he used them for navigation back in his smuggling days. He also refers to some of the constellations by name.
A half moon was sliding in and out amongst thin high clouds, and Davos could see familiar stars. There was the Galley, sailing west; there the Crone's Lantern, four bright stars that enclosed a golden haze. The clouds hid most of the Ice Dragon, all but the bright blue eye that marked due north. The sky is full of smugglers' stars. They were old friends, those stars; Davos hoped that meant good luck.
A Storm of Swords, Davos VI
Davos tells of how he knows the shapes of the coastlines, most likely having had sailed around them enough times and used them for navigation purposes.
"There's much I don't understand," Davos admitted. "I have never pretended elsewise. I know the seas and rivers, the shapes of the coasts, where the rocks and shoals lie. I know hidden coves where a boat can land unseen. And I know that a king protects his people, or he is no king at all."
A Storm of Swords, Davos VI
Tyrion commands the captain who is taking Myrcella to follow the coastline, though it's not clear if this is for protection or navigation.
He cleared his throat. "You know your orders, Captain."
"I do, my lord. We are to follow the coast, staying always in sight of land, until we reach Crackclaw Point. From there we are to strike out across the narrow sea for Braavos. On no account are we to sail within sight of Dragonstone."
A Clash of Kings, Tyrion IX
The following quote tells of a priest telling Victarion to use the coastline to navigate to find some ships to pillage.
"The captain commands, and I obey," said Moqorro. The crew had taken to calling him the Black Flame, a name fastened on him by Steffar Stammerer, who could not say "Moqorro." By any name, the priest had powers. "The coastline here runs west to east," he told Victarion. "Where it turns north, you will come on two more hares. Swift ones, with many legs."
A Dance with Dragons , Victarion I
Sailors also use Myrish lenses/far eyes which is essentially a telescope. These would have been used for various reasons such as scouting the enemy but they are also used for spotting land. As can be seen when Catelyn sails for King's LKanding.
High overhead, the far-eyes sang out from the rigging. Captain Moreo came scrambling across the deck, giving orders, and all around them the Storm Dancer burst into frenetic activity as King's Landing slid into view atop its three high hills.
A Game of Thrones, Catelyn IV
Note in the above quote "far-eyes" is referring to the person who is using the far eyes and sitting in the crows nest. So, just to be complete here is a reference to a captain owning one.
This time it was a Myrish cog named Dove, on her way to Yunkai by way of New Ghis with a cargo of carpets, sweet green wines, and Myrish lace. Her captain owned a Myrish eye that made far-off things look close—two glass lenses in a series of brass tubes, cunningly wrought so that each section slid into the next, until the eye was no longer than a dirk. Victarion claimed that treasure for himself.
A Dance with Dragons, Victarion I
It is also evident that sailors use landmasses for navigation in that they use maps and charts.
Thereafter, Ghiscari sailors took care to stay well away from the Demon Isle, as they named Walano on their charts; they had no inkling of the existence of Omboru, Jhala, or the lesser islands.
Koj, once home to Malthar the Mapmaker, still boasts the finest shipyards in the archipelago. Threequarters of the islanders' famed swan ships are built on Koj, and the Pearl Palace, seat of the Princes of Koj, is renowned for its collection of charts and maps.
The World of Ice and Fire, Beyond the Free Cities: The Summer Isles