Possibly that scene invokes Bran the Blessed, the giant king in the Welsh Mabinogion. Bran waded across the sea with his fleet. There was a cauldron in which the ogres were going to cook the Time Bandits. A cauldron was also central to Bran's story, although that one brought the dead back to life.
Because it was a transition from ogres to a giant, the scene might also have Norse overtones. Frost giants and ogres are abundant in Norse mythology. Perhaps the scene reflects the Viking incursions in England. The giant crushing the hut might be Viking raids (carried out on longships) and its strides across the land might reflect the centuries of time over which these incursions lasted. In 1066, the English pushed them back at last...only to be defeated by the Normans...another Norse offshoot. Ultimately, the Norse/Vikings were subsumed into the culture...a sleeping giant that had left its mark on the history of the British Isles.
Or...the scene didn't have any particular relation to legends or history and was just Gilliam's imagination in action.