The Targaryen army withdrew from Dorne, leaving the country untaken. A thousand pinpricks did what a single pitched battle could not, and Aegon chose to withdraw and leave Dorne an independent nation.
A Wiki of Ice and Fire, War of Conquest
Why did Aegon give up? What kind of tactics were used by House Martell to make this happen?
It wasn't that Aegon didn't want to capture it. He couldn't, because the defendants fought in a way Aegon could not use his dragons. They knew that they didn't stand a chance in open battle. They were using ambush techniques and were not hiding in castles like other kings did.
Whereas other kings and lords had taken to the field against Aegon, or clustered in castles, the Dornish refused to give open battle and allow Aegon to deploy his dragons. Instead, they turned to ambush and raids, striking quickly and then slipping back into the desert or through the mountain passes, where even the dragons could not find them. In time, Aegon pulled away from Dorne.
(A Dance with Dragons, Chapter 8, Tyrion & A Feast for Crows, Chapter 28, Cersei)
Accepted answer is correct but lacking in details.
This has been discussed in detail in the new book The World of Ice and Fire. Since when previous answer was given, this information had not been revealed so I want to add it here.
I will split the answer in different parts as per important events in the failed invasion of Dorne.
Quasi-Invasion of Dorne and Vale started at proximately the same time. Visenya rode Vhagar to Vale and Rhaenys rode Meraxes to Dorne to negotiate their surrender and give them a last chance of submitting before unleashing the dragons on them. It was Quasi-invasion because the expected outcome was submission of the said Kingdoms after show-of-power in form of Dragons flying over their heads but at the same time, these missions were unaccompanied by traditional invading forces.
Visenya won a bloodless victory when she took King of the Vale to fly with her on her dragon.
Rhaenys did not have such an easy time however. From Land route, there are two ways of entering Dorne.
Rhaenys took the Prince's pass route. She saw a Dornish Army guarding the pass but she did not engage them and instead decided to fly directly to castles of the Lords and force their submission without shedding blood.
Rhaenys first flew to Vaith but found the Castle empty. She then moved to Godsgrace but that castle was abandoned as well. Upon inquiring, Peasants only told her that Lords were gone to an undisclosed location. Rhaenys then moved East to Planky Town, chief port of Dorne. She found only women and children there and all the men were gone.
At that Rhaenys decided to visit Sunspear directly. When she reached there, Sunspear was empty as well but the Princess of Dorne, Meria Martell was present there. Following conversation took place between the Princess and the Queen:
“Meria: I will not fight you, nor will I kneel to you. Dorne has no king. Tell your brother that.
Rhaenys: I shall, but we will come again, Princess, and next time we shall come with fire and blood.
Meria: Your words, Ours are Unbowed, Unbent, Unbroken. You may burn us, my lady, but you will not bend us, break us, or make us bow. This is Dorne. You are not wanted here. Return at your peril."
Meria made it clear for Rhaenys that they had no intention to fight the Targaryens but they will never submit either. At that Rhaenys went back to her brother who had just taken Old Town and was crowned there as "King of the Andals and The Rhoynar and the First-men, Lord of the Seven Kingdoms". So as evident by the titles, Aegon certainly wanted to conquer Dorne as he proclaimed himself King of the Rhoynar and Lord of the Seven Kingdoms.
Pictured, Queen Rhaenys meeting Princess Meria Martell at Sunspear.
Aegon left Dorne alone for sometime while he established his rule in King's Landing.
He was done with those administrative tasks and minor wars in four years so in 4 A.C. he began full invasion of Dorne to finish the conquest.
Dornishmen resorted to fighting a guerrilla war instead of facing the Targaryens in a pitched battle. They abandoned their castles since they knew what happened to King Harren the Black in burning of Harrenhal and fled into Dornish wilderness which was composed mountainous and desert terrains, perfect for this sort of fighting. Not to mention, Dornishmen were fighting on home ground while invaders were fighting in hostile terrain with terrible weather for those unaccustomed to heat.
Rhaenys struck the first blow. She flew along the coastline and struck at Planky Town. She burnt the whole city and lead her men Northwards to Sunspear. I am not sure about the type of force she commanded but it was presumably an amphibious force because her mission was to attack Dorne from sea and strike towards Sunspear.
In the Meanwhile, Aegon lead an army from Prince's Pass into Dorne. Lords of Red Mountain did not offer regular resistance to Aegon. As Aegon was making his way to Hellholt, Lords of Red Mountain performed hit and run attacks on Invaders, causing huge damages. Whenever Aegon or Visenya mounted their dragons, the Dornish Fighters disappeared in the terrain. When finally Aegon reached Hellholt, he found that House Uller who ruled the land were all gone before he could capture the castle.
Annoyed, Aegon moved North-West to Yronwood. He took the castle but was further annoyed when he found that Castle was defended by old men and children while fighting men were all gone. When Aegon reach Skyreach, he found that the castle of house Fowler was abandoned. So Aegon considered his quest of conquering Red Mountains as completed and move Eastwards along the coast to take Ghost hill, Stronghold of House Toland.
As soon as Aegon besieged Ghost Hill, a challenger in livery of House Toland challenged Aegon to single combat. Aegon made short-work of the challenger, expecting him to be Lord Toland. However, when the corpse was identified, It was revealed that the Challenger was actually fool of Lord Toland while the Toland Family had already fled.
Aegon had planned to invade Dorne from all sides. So command to attack from Boneway was Orys Baratheon's. Orys suffered a disastrous defeat when Dornish men attacked the Targaryen Army in a surprise Night attack. Targaryens were trapped. Many notable Lords were taken POWs by the Dornish which included Hand of The King, Orys Baratheon, The man rumoured to be bastard brother of Aegon, Rhaenys and Visenya (Confirmed in Fire and Blood).
Still, save the assault from Boneway, the other two Targaryen invading forces were advancing successfully. There was little to no resistance as castles were abandoned and soldiers were vanished. Soon, Rhaenys and Aegon joined each other at Sunspear where they found the seat of House Martell empty. Aegon declared himself as victor of the Dornish war and named Lord Rosby as Royal Viceroy of Sunspear. He also stationed an Army in Dorne under command of Lord Tyrell to put down any revolts as Aegon knew that power of House Martell was still intact as they had not fought in any major confrontation.
Content with the outcome and confident about Lord Tyrell's capabilities, Aegon and his sisters left Dorne for King's Landing.
As soon as Dragons left, Rebellion spread like wildfire as that was the Dornish plan all along. Martell Forces rallied and the war finally began in earnest.
Lord Tyrell and his entire army disappeared somewhere between Hellholt and Vaith, presumably destroyed by Martells.
Victorious Martells then moved to liberate Sunspear. The Targaryen Viceroy (Officially Warden of the Sands), Lord Rosby was arrested. Dornishmen took him to the roof of Spear Tower and blind eighty years old Meria Martell personally threw him down to his death.
Martells offered to free the POWs for ransom. Aegon consented as his Hand, Best Friend and bastard half-brother Orys Baratheon was among the captives. Dornishmen demanded weight of each man in gold as ransom. Aegon agreed and ransom was paid. However, Dornishman Lord Wyl cut off sword hands of the captives before freeing them (Orys eventually avenged himself on Lord Wyl's son by taking his sword hand as payment and the other hand and both feet as his usury).
Aegon was enraged. Dornishmen had broken the agreement of release of Prisoners by mutilating them. Aegon and his sisters burnt many Dornish cities in revenge. In retaliation for Air-attacks, Dornishmen launched raids in Reach and Stormlands. Aegon burnt Starfall, seat of House Dayne. Rhaenys went to burn Hellholt, seat of House Uller. Hellholt was burnt but an ironbolt from a scorpion pierced eye of Queen Rhaenys' dragon Meraxes and she fell. She was presumed dead as she took quiet a tumble there.
Aegon, Orys and Visenya were devastated by death of Rhaenys. The next two years were called Dragon's wroth due to wanton brutality and destruction showed by Targaryens to avenge Rhaenys.
Aegon and Visenya burnt every Dornish castle except Sunspear. Some people say Aegon and Visenya had come to fear that Princess Meria had somehow figured out how to bring down Dragons therefore they avoided attacking Sunspear. An alternate reason is that Aegon had hoped if Dorne was to burn and Sunspear was to remain intact, it would turn Dornishmen against Martells. Aegon launched a campaign of Propaganda that Martells had actually paid the Targaryens to spare their castle while decimating rest of Dorne. Dornishmen however rejected this propaganda and remained loyal to Martells.
Still unsatisfied, Aegon and Visenya declared the Dornish lords to be rebels and outlaws and placed bounties on their heads. Dornish responded by placing bounties on heads of the Targaryen siblings.
Aegon and Visenya came very close to be assasinated but Visenya's quick thinking and sword skills saved them. Then Visenya founded the order of King's Guard to protect the King from Dornish assassins.
In the meanwhile, Martells committed atrocities in Stormlands.
After 9 years of warfare, Princess Meria Martell died. A Man of peace, Her Son, Prince Nymor Martell sent a peace delegate under his daughter the Princess Deria. He also sent skull of Meraxes though it is unclear whether it was sent as a warning or as a gesture of goodwill.
Nevertheless, The Skull was seen as an open threat in King's Landing and response was outrage specially by powerful leaders such as Orys Baratheon and Visenya Targaryen. Aegon was about to refuse Deria's peace terms but then the Princess produced a letter from her father which was for the King's Eyes only.
No one knows the contents of the letter but Aegon's color fled when he read it and he cut himself on the throne. Some people suggest that the letter said Rhaenys was alive but less than dead. Martells had tortured Rhaenys into insanity and offered to end her misery and return her corpse if Aegon was to give them peace.
Aegon left King's Landing for sometime, presumably to receive corpse of Rhaenys. When he returned, he signed the peace agreement with the Dornish.
Reasons of failure should be evident from the complete account of the invasion. In any case, the reasons of Aegon's failure and lack of further attempts appear to be:
