In book 1 of Harry Potter we learn that Voldemort is undone the night before Harry is left with the Dursleys. This is gleaned from the exchange between Professor McGonagall and Dumbledore outside the Dursleys' house while they wait for Hagrid.
‘What they’re saying,’ she [Professor McGonagall] pressed on, ‘is that last night Voldemort turned up in Godric’s Hollow. He went to find the Potters. The rumour is that Lily and James Potter are – are – that they’re – dead.’
Shortly after this Hargid arrives in his motorcycle, making it sound like he came directly from getting Harry out of his parent's house.
‘No, sir – house was almost destroyed but I got him out all right before the Muggles started swarmin’ around. He fell asleep as we was flyin’ over Bristol.’
Now Google Maps places Bristol a little over 2 hours west of Surrey, following the roads, and while the exact location of Godric's Hollow is is unknown the HP Lexicon places it in West Country. West Country typically includes Cornwall, Devon, and Somerset, which places the location of Godric's Hollow somewhere South, South-west of Britstol, at most somewhere between 2 and 3 and hours away (again by road).
So in a flying motorcycle, following a route that would take them over Bristol, and assuming Hargid was going somewhat slow so as not to scare or wake Harry, it might take between 6 and 8 hours to get from the Potter's house to the Dursleys' house. So where were Hagrid and Harry in the extra 10 to 18 hours (since Hargid says he got Harry just as the muggles began to arrive, which should have been fairly fast)?
The hedge had grown wild in the sixteen years since Hagrid had taken Harry from the rubble that lay scattered amongst the waist-high grass. Most of the cottage was still standing, though entirely covered in dark ivy and snow, but the right side of the top floor had been blown apart; that, Harry was sure, was where the curse had backfired.