While I would normally agree that Tolkien frequently uses the word "doom" in the archaic sense, meaning something like "destiny, judgement, fate" etc, I don't think that is the case in this particular instance. I think Treebeard genuinely believed that he and the other Ents were going to be killed in the battle.
But another peculiarity of Tolkien comes into play here: We generally hear what the individual characters think about things, and almost never what Tolkien, as the narrator and "voice of god", thinks about things. This is an example of Treebeard saying what he believes, but being wrong.
It was, of course, quite possible that the Ents wouldn't have done so well in the battle, and that many of them- perhaps even all of them - would have died. The Ents are certainly powerful, but a wizard is an extremely formidable adversary, even for the Ents. Treebeard was wrong about going to his doom, but I'm sure he was happy to be proven wrong.