If you're looking for the "canon-ish" answer for the change, according to the "Star Trek: The Next Generation Technical Manual" the scale was recalibrated in the 24th century. The old scale was calculated based on "observed" speed (much like our MPH/KPH), but the amounts of energy needed to maintain that speed could be vastly different from one moment to the next based on interstellar conditions and quantum drag forces. For instance you spend a few seconds traveling through a smallish eddy and it greatly reduces your velocity. Since such great distances were being covered in such a short time, a great deal of turbulence exists. Since the engines aren't actually fluctuating power to maintain a constant speed, the observed (averaged) speed was little more than a guess. I imagine this like driving a car at 6000 RPM over a surface that is at one second oil and the next sandpaper, with large rocks mixed in here or there. You could certainly figure out what your speed was for the last mile, but it may not be indicative of your speed for the next mile.
In TNG, the scale was changed to the amount of power required to transition from one warp plateau to another. The idea here is that it's easy (power-wise) to maintain a particular speed, but a power spike is needed to make that jump from Warp 1 to Warp 2. A larger spike is needed to get from Warp 1 to Warp 9. At that time, they decided Warp 10 was infinite power required. So all the "undiscovered" plateaus between 9 and 10 had to be squeezed in. I liken this to the metric system, where they decide a gram was a rather small amount of mass, but rather than recalibrate, people just decided to measure things in kgs.
Outside of canon, I recall an interview with Roddenberry. He put in a speed limit to keep writers from inventing more speed as the way out of trouble. You can see this a few times in the first season where the Enterprise tries to outrun things, but cannot (Qs grid springs to mind).
Edit: Just found online link to the TNG Technical Manual with a note alluding to Gene's decision. Check page 55.