Were there any marriages before the beginning of the book "Game of Thrones" between any Tyrell lord and any woman from the Royal or ex-royal families (for example, Starks, Lannisters, Targaryen, Durrandons, Gardeners etc)?
From the Wiki:
The Tyrells were stewards to House Gardener, the ancient ruling line of the Kings of the Reach, who periodically intermarried with lower ranking houses of the Reach such as House Tyrell or House Florent.
I don't know of any other examples but this means that the Tyrells had marriages to House Gardener which was a royal familiy
Tyrells and Lordship status
Your question is about Lords of House Tyrell. House Tyrell gained lordship after Aegon's conquest and Field of Fire. Before that, they were stewards to the Kings of the Reach who belonged to House Gardener.
The era of minor Andal and First men Kings
But one thing that you have not mentioned is, there are many minor Houses in Westeros who used to be Kings in their own right before Starks, Lannisters, Gardeners, Durrandons, Martells forced them into submission. Examples of these former Royal Houses are House Yronwood, House Dayne, House Bracken, House Blackwood etc.
Historical Lords of House Tyrell
Historically, We do not know many Tyrell Lords or their consorts. The only Lord Tyrells that have been mentioned are:
- Lord Harlen Tyrell - He was the first Lord Tyrell. He was descended matrilineally from Garth Greenhand. Garth Greenhand was patrilineal legendary ancestor of most Houses in the Reach and a King. So if Tyrells are descended from his female line, it is logical that there must have been some marriage between a Tyrell male and a royal female however Tyrells wouldn't be lords back then and it is not necessary that the woman would most certainly be a Princess. She could be a distant cousin of the Royal House like most of the aristocracy of Reach. In any case, Lord Harlan's spouse is unknown as of now.
- Lord Theo Tyrell - His Spouse is not mentioned.
- Lord Mathos Tyrell - Lady wife is not mentioned. He attended Grand Council of 101 AC.
- Unnamed Lord/Lordling Tyrell (Plausibly son of Lord Mathos) - He tried to court Princess Rhaenyra Targaryen in 112 AC when his father was still Lord of the Reach but the courting failed as the Princess married Laenor Velaryon instead. Name of this certain Lordling's father is not mentioned either but since there is only a difference of 11 years between GC of 101 and Rhaenyra's visit of High Garden, It is very possible that He may have been son of Lord Mathos. It is however not mentioned whether he ever became Lord Tyrell or not because the ruler of Reach during Dance of Dragons in 129 AC was Lord Lyonel Tyrell who was an infant at that time. So Lord Lyonel may have been this character's son or younger brother.
- Lord Leo Longthorn - His spouse is unknown.
- Lord Luthor Tyrell - His spouse is Olenna Redwyne. Redwynes used to be Kings until their Last King was lost in the sea and the remaining Redwynes swore fealty to King of the Reach Meryn III Gardener and became a part of their realm.
So from the existing sources, yes there has been a marriage between a Tyrell Lord and female from a former Royal Dynasty.
The Marriage between Luthor Tyrell and Olenna Redwyne fits this bill.
Inter-kingdom marriage was not common before the timeline of the books. Lords Paramount would typically marry their children to vassals in order to keep vassals loyal.
Rhaegar's marriage to Elia Martell was only because there were no Targaryens available to marry. Even then, Rhaegar's cousin went first to Essos to find suitable brides of the blood of Valyria without success. Elia was the last choice.
The only reason it seems common for Lords Paramount to intermarry their families is because it was done to seal alliances for Robert's Rebellion. The Riverlands wouldn't have joined the war without wedding its daughters to Stark and Arryn.
The practice continued after the war for two reasons. Marriage alliances were powerful tools now proven to be effective and House Baratheon didn't marry brother to sister like the Targaryens did.