Probably some supernatural bond
This promise that Dormammu makes (not to attack Earth directly) is taken straight from the comic universe, though the context is different.
In the comics, Dormammu has a rigid morality. According to Strange:
STRANGE: He is evil, true…but only by our human standards. According to his
own lights, he has his own moral code!
Whether there is something supernatural to this is unclear. It seems very likely, though. Other media (not canon to the comics or movies, naturally) do imply as much. For example, the Infernal Promise card from Marvel’s War of Heroes card game.
Mortal! Do not take lightly the oaths of Dread Dormammu! There is
power in the promises of beings such as myself! I will honor any
agreement between us! But a simple misstep on your part might undo our
arrangement! And when that happens, Dormammu will owe you no such
thing as mercy!
More generally, supernatural beings are often depicted as being forced to keep their word: Faeries not lying, Greek gods taking oaths of the River Styx, deals with the Devil.
Would Strange know this, though? Probably. He has been studying magic for some time, and probably knows the rules that bind entities such as Dormammu. Even if he doesn’t, can we be sure he’d really be willing to be stuck in a loop for all eternity? Perhaps he’s simply taking a chance, and hoping Dormammu keeps his promise (as he does in the comics).
Then, too, Strange still has the Time Gem. If Dormammu tries again, it’s quite possible that Strange would just put him back in the loop. That would serve as a strong deterrent, even if Dormammu is not magically bound to his promise.