I think there are two possible answers.
Let’s start with this passage:
“They know!” whispered Griphook in Harry’s ear. “They must have been warned there might be an impostor!”
“Your wand will do, madam,” said the goblin. He held out a slightly trembling hand, and in a dreadful blast of realization Harry knew that the goblins of Gringotts were aware that Bellatrix’s wand had been stolen.
“Act now, act now,” whispered Griphook in Harry’s ear, “the Imperius Curse!”
— Deathly Hallows, chapter 26 (Gringotts)
Hermione is using Bellatrix’s original wand, which they stole in the breakout from Malfoy Manor. Of course, the real Bellatrix will have reported this as stolen, and so it would be highly unusual for her wand to show up at Gringotts. Although Harry imperiuses Bogrod (the goblin on the desk) and Travers (the Death Eater accompanying them), their story will fall apart under any scrutiny.
There are then two possibilities:
Another goblin asks about the wand. The trio get their attention when they approach the desk, and Hermione asks to enter the Estrange vault:
The old goblin seemed to recoil a little. Harry glanced around. Not only was Travers hanging back, watching, but several other goblins had looked up from their work to stare at Hermione.
If any of them watch Bogrod ask for an identifying wand, and suddenly accept her story, they’re bound to be suspicious. Especially since Bogrod and Travers start acting a little strange: that alone may move them to test the Thief’s Downfall on their intruders.
(I’m assuming that had they been legitimate visitors, they would have passed through the Thief’s Downfall unchanged and then, their purpose confirmed, been rerouted to their vault.)
The Imperius curse wears off Travers. Harry casts the Imperius curse on both Travers and Bogrod, but only Bogrod comes down into the vault. Harry sends Travers to hide, but expresses doubts about his Imperius Curse:
“They’re Imperiused,” he added, in response to Hermione and Ron’s confused queries about Travers and Bogrod, who were both now standing there looking blank. “I don’t think I did it strongly enough, I don’t know….”
If Travers wakes up and realises he’s been Imperiused, he’ll probably run straight out and tell the goblins what’s going on, blowing their cover.