I just noticed the wording on the quote about the Fidelius charm from PoA, it specifically mentions soul, which as we know is a big part of the "magic system".
An immensely complex spell involving the magical concealment of a secret inside a single, living soul. The information is hidden inside the chosen person, or Secret-Keeper, and is henceforth impossible to find — unless, of course, the Secret-Keeper chooses to divulge it. As long as the Secret-Keeper refused to speak, You-Know-Who could search the village where Lily and James were staying for years and never find them, not even if he had his nose pressed against their sitting room window!"
—Filius Flitwick explaining the charm
in the context of creating a Horcrux, or avada kedavra, magic that interacts with the soul seems fairly permanent, especially from Voldemort's experience of being hit by the killing curse, his soul is ripped from his body, and is only protected by the existence of the Horcrux's, and returned to the body by powerful dark magic.
“I was ripped from my body, I was less than spirit, less than the meanest ghost . . . but still, I was alive.”
― Voldemort, Harry Potter and the Goblet of Fire
I believe for this reason, the removal of the Fidelius charm would have been immensely damaging to the soul that kept the secret, and in the event of the death of Dumbledore, I speculate that the action would have needed to be performed on each of the new secret keepers.
Also as others have mentioned, there is no reference to the removal of the charm throughout the books, and I doubt it would have even been attempted, or created as a spell.
As we can see from most other spells from the universe, a lot of spells do not have a reversal i.e Lumos and Nox, however they have counter magic/spells, or magic/spells to fix the damage, and in this case, I'd guess the reversal is just not possible.
Hope this answers your question.