We're not told in the text exactly how Sauron was driven out of Dol Guldur. The Hobbit only says:
It appeared that Gandalf had been to a great council of the white wizards, masters of lore and good magic; and that they had at last driven the Necromancer from his dark hold in the south of Mirkwood. "Ere long now," Gandalf was saying, "The Forest will grow somewhat more wholesome."
(The Hobbit, Chapter 19, "The Last Stage")
The Fellowship of the Ring doesn't add much more. Gandalf merely says:
"Some, too, will remember also that Saruman dissuaded us from open deeds against him, and for long we watched him only. Yet at last, as his shadow grew, Saruman yielded, and the Council put forth its strength and drove the evil out of Mirkwood. ... It was by the devices of Saruman that we drove [Sauron] from Dol Guldur."
(The Fellowship of the Ring, Book II, Chapter 2, "The Council of Elrond"; emphasis added)
And that's all we're ever told.
With this in mind, it's not even clear that there was a physical confrontation between the White Council and Sauron, much less that they were physically present—indeed, I would argue that the phrasing "put forth its strength" suggests that they were not physically present. As importantly, we're only told that Sauron left; we don't know that there weren't others of his servants still present.
Thus, though we're never told exactly why Dol Guldur wasn't destroyed, we can raise the following possibilities:
The White Council wasn't physically present at Dol Guldur, and thus could only affect Sauron, not the fortress itself.
The Council's action affected only Sauron, and wasn't applicable to his servants, who held the fortress.
Both of the above.