The temple was no longer functional as a weapon once Ezra removed the holochron powering it.
Remember, Vader clearly wasn't expecting Ezra and Kanan to sneak inside and grab it while he fought Ahsoka, and indeed, later on he attempted to recover it from Ezra's hand but was ultimately unsuccessful.
Now back to the temple itself, what Ezra did is basically equivalent to pulling the plug on a running PC—yes, it turns the thing off but often results in unintended side effects. In this case, removing the holochron disrupted the flow of energy within the temple, initiating the collapse. This disruption likely also damaged the main kyber crystal, which could have been the cause of the large explosion seen as the Phantom is flying away:
This is consistent with other kyber-related explosions we've seen in other places. In particular, the more recent episode Holochrons of Fate (season 3, episode 2) shows us another example, when
Maul and Ezra combine the Jedi and Sith holochrons, ultimately resulting in an explosion that "destroys" both of them.
Ezra is literally feet away when this happens, but neither he nor his companion are physically injured, nor is there any damage to the room. Given this, it's not at all surprising that Ahsoka and Vader survived the blast—the explosion is likely more representative of the breakdown of whatever Force-based metabehaviors are embedded in those objects. Consider the fact that:
The pieces of the destroyed holochrons are still there and intact, but they no longer "stick" together:
So in conclusion, yes, the temple was still there, but it was no longer in actual working condition.