This has not currently been explained definitively in the TV show and I cannot speak for the books, as i have not read them.
As seen in the TV show, S5E8 'Hardhome', the corpses of all of the dead wildlings and brothers of the watch do not reanimate until the Night King raises his hands to force them to. This suggests he can convert them at will.
It may be possible that they were intended not to reanimate until they were back at castle black in the warm, on purpose, as a form of 'sleeper' attack to cause chaos and kill some brothers of the watch, or perhaps until the Night King expressly commands them to rise, they do so on their own, albeit slower.
This is only a theory, however. It has not been explained by the showrunners, writers or actors.