I interpret the "lateness" as meaning that Clara's presence made it arrive 300 years later than it was supposed to.
By the time she latched onto it, it had already started to disappear, presumably with coordinates set for Trenzalore 300 years earlier than it actually arrived. With an unplanned passenger on the outside, the calculations for getting it to the right place and time were slightly off, and so it arrived at the wrong time (300 years late).
CLARA: I was in space.
DOCTOR: Well, you were in the Time Vortex. She must have extended the force field. No wonder. No wonder she's late, dragging you around.
How do you even define "speed" in the Time Vortex? Being 'late' while travelling through time makes about as much sense as visualising time travel using a picture of a physical vortex.
We see the same effect when Jack Harkness hitches a lift on the outside of the TARDIS in the series 3 episode Utopia. In this case, his unexpected arrival causes the TARDIS to move way way into the future, to the very end of the universe.