I am not able to remember the name of this anime. There were yellow monsters, and the people who fought them used big machine suits. The more they used the suits, the more they turned into crystals and eventually died. It was set on an island, and there was a girl on the island who suddenly became full grown from a child.
Fafner in the Azure (2004), probably.
Tatsumiyajima is the central island in the middle of a small cluster of islands, in a sleepy backwater of the Japanese isles. Not much happens there, and the island's young people go to school knowing that their lives are likely to remain peaceful and undisturbed. Or so they have been taught... but the truth is different. The fate of mankind is on the line, and Tatsumiyajima is the last line of defense against a hostile and incomprehensible enemy. At the center of it all, fighting for Humanity's continued existence, is the giant robot Fafner, the dragon that guards this final treasure of mankind.
The yellow monsters would be the Festum:
The Festum have an "assimilation" process, by which the target gets crystallized, shattered and incorporated to their hive mind (the Mir). The Fafner (the big mecha humans use to fight Festum) use Mir components, and that causes what you remembered: (info taken from the Fafner Wiki)
Due to the use of Mir fragments in the construction of many Fafner models, their internal cores gradually assimilate pilots with each activation, in largely the same manner as Festum. The rate of assimilation varies wildly between models, but given time, every core-equipped Fafner will inevitably attempt to consume its pilot. [...]
In the worst of cases, the Fafner will begin attacking its pilot's mind, leaving them comatose and unable to continue piloting. Without proper treatment, comatose victims or other severe cases will eventually succumb to complete assimilation, their body crystallizing entirely and shattering, just as Festum victims do.
See this woman pilot for instance (S01E23, "Decoy"):
And the opening for good measure:
Found with the Google query
anime mecha "turn * crystal" -"guilty crown" which returned this answered Yahoo question, which was likely searching for the same thing:
What was that mecha anime where they would turn to crystal when they were defeated? If i remember right they were on a moving island to prevent them from being located. I just remember that when they lost they would have crystals growing all over there mech's