There's a reference to La Crau (or rather the plain of La Crau) in the catchily titled 'Remarks on several parts of Europe, relating chiefly to their antiquities and history : collected upon the spot in several tours since the year 1723 and illustrated by upwards of forty copper plates, from original drawings. Vol. 2' by John Durant Breval.
In short, a local tale tells of Hercules coming upon two giants, the sons of Jupiter (Zeus); Albion and Borgion. Hercules fought them until his arrows were spent, at which point Jupiter sent him a shower of rocks which he used to defeat the two giants. There's no specific mention of what they were up to, but building a tower wouldn't be out of the question.
The inscription on the town fountain in nearby Arles tells a similar story (in translation)
"To leave the island of Eritrea, Heracles again embarks, with the oxen
he conquered, on the cup of the sun, then returns after having
approached the continent. He continues his journey across the European
continent, pushing the flock before him; first he crossed Iberia, then
Celtic, where Alesia is a foundation of the hero; in the deserted
region of La Crau, east of the Bouches du Rhone, he has to undergo a
terrible struggle against the Ligyens; his arrows being exhausted and
himself on the verge of fainting with fatigue, Zeus dropped a shower
of stones which served him to complete the victory. He crossed the
Alps, repulsed in Liguria the attack of two sons of Poseidon, Alébion
and Derkymos, who want to rob him of his cows, crosses Tyrrhenia,
arrives at Rome, where he triumphs over Cacus, arrives in the region
of Cumae, where he supports a fight against the Giants in the
Phlegraean fields, builds the causeway that once separated from the
sea Lake Avernes and Lucrins. In the vicinity of Rhegium, one of his
bulls escapes and drives him in pursuit to Sicily, which is full of
his exploits and memories that left his passage. "
It may interest you to learn that this story has changed considerably in the retelling. The earliest reference I can find is from the Geography of Strabo written circa 20 B.C in which he pours scorn on the idea that the rocks were heaven-sent (an account that he attributes to the fanciful imagination of an earlier geographer, Æschylus) and that they were most likely a geological feature relating to the movement of water.
He also mentions an earlier (and presumably more accurate account) of a skirmish that took place on the plain where the attackers used the stones to their advantage, presumably in slings or small siege engines. This may be the root of the story or it may be wholesale an invention of Æschylus, it's not clear.
The whole both of this district and that above it is exposed to the
wind, but in this plain the black north, a violent and horrible wind,
rages especially: for they say that sometimes the stones are swept and
rolled along, and men hurled from their carriages and stripped both of
their arms and garments by the force of the tempest. Aristotle tells
us that these stones being cast up by the earthquakes designated
brastai, and falling on the surface of the earth, roll into the
hollow places of the districts; but Posidonius, that the place was
formerly a lake, which being congealed during a violent agitation,
became divided into numerous stones, like river pebbles or the stones
by the sea-shore, which they resemble both as to smoothness, size, and
appearance. Such are the causes assigned by these two [writers];
however, neither of their opinions is credible, for these stones
could neither have thus accumulated of themselves, nor yet have been
formed by congealed moisture, but necessarily from the fragments of
large stones shattered by frequent convulsions. Æschylus having,
however, learnt of the difficulty of accounting for it, or having been
so informed by another, has explained it away as a myth. He makes
Prometheus utter the following, whilst directing Hercules the road
from the Caucasus to the Hesperides: “‘There you will come to the
undaunted army of the Ligurians, where, resistless though you be, sure
am I you will not worst them in battle; for it is fated that there
your darts shall fail you; nor will you be able to take up a stone
from the ground, since the country consists of soft mould; but
Jupiter, beholding your distress, will compassionate you, and
overshadowing the earth with a cloud, he will cause it to hail round
stones, which you hurling against the Ligurian army, will soon put
them to flight!’”
Posidonius asks, would it not have been better to have rained down
these stones upon the Ligurians themselves, and thus have destroyed
them all, than to make Hercules in need of so many stones? As for the
number, they were necessary against so vast a multitude; so that in
this respect the writer of the myth seems to me deserving of more
credit than he who would refute it.