I've heard two primary theories:
- Frollo has completely lost his sanity.
- The gargoyle is alive and passing judgement on Frollo.
Both of these theories tie into the rest of the movie neatly. This is especially because they relate to the parallel discussion of whether Hugo, Victor, and Laverne, Quasimodo's three gargoyle friends, are truly alive or merely Quasimodo's imagination. If Quasimodo invented the gargoyles out of his loneliness and isolation, then Frollo losing his mind to the point of having similar hallucinations makes his comeuppance that much more karmic.
By another interpretation, there are reasonable thematic reasons to believe the gargoyle Frollo saw was indeed alive. Hugo, Victor, and Laverne are treated ambiguously throughout the whole movie, so animate gargoyles are plausibly part of the story's setting. And symbolically, the Cathedral of Notre Dame is hinted to have a personality and force of will all to its own. At the beginning of the film, Frollo refuses to spare baby Quasimodo's life until he realizes the church's statues of the saints are staring at him. The animation and camerawork used makes the statues appear to be sentiently judging Frollo's actions. By this interpretation, the gargoyle that breaks on Frollo is another statue in the church judging his sins, deliberately condemning his soul. This is a strong symbolic and motivic bookend with the statues holding Frollo accountable to his murder at the film's opening.
I don't see much evidence to believe the gargoyle is an incarnation of a devil coming to take Frollo's soul. The imagery of Satan attacking humans is used in only one place in the film, during Frollo's "Hellfire" song, when the judge says, "God made the devil so much stronger than a man." Frollo says this as a way of trying to reject his responsibility for his sinful lust, so his reference to Satan is deliberately theologically flawed and dishonest. So this reference to Satan is less about establishing Satan as an important symbol or character in the story and more about revealing Frollo's heart.
Within this film, God is portrayed as the one who sends sinners to hell, not Satan. Consider again the scene with the statues silently judging Frollo. And throughout the movie, with the only unhinged exception above, Frollo consistently describes himself as carrying out God's judgement on sinners, sending them to hell where they righteously deserve to be.
All this considered, I find it doubtful that demons show up to claim Frollo's soul at the end of the film. They haven't showed up anywhere else in the film, and the only reference to them in the film comes from an unreliable character during one of his most thoroughly unhinged moments.