If you're looking for the "canon-ish" answer for the change, according to the "Star Trek: The Next Generation Technical Manual" the scale was recalibrated in the 24th century. The old scale was calculated based on "observed" speed (much like our MPH/KPH), but the amounts of energy needed to maintain that speed could be vastly different from one moment to the next based on interstellar conditions and quantum drag forces. For instance you spend a few seconds traveling through a smallish eddy and it greatly reduces your velocity. Since such great distances were being covered in such a short time, a great deal of turbulence exists. Since the engines aren't actually fluctuating power to maintain a constant speed, the observed (averaged) speed was little more than a guess. I imagine this like driving a car at 6000 RPM over a surface that is at one second oil and the next sandpaper, with large rocks mixed in here or there. You could certainly figure out what your speed was for the last mile, but it may not be indicative of your speed for the next mile.
In TNG, the scale was changed to the amount of power required to transition from one warp plateau to another. The idea here is that it's easy (power-wise) to maintain a particular speed, but a power spike is needed to make that jump from Warp 1 to Warp 2. A larger spike is needed to get from Warp 1 to Warp 9. At that time, they decided Warp 10 was infinite power required. So all the "undiscovered" plateaus between 9 and 10 had to be squeezed in. I liken this to the metric system, where they decide a gram was a rather small amount of mass, but rather than recalibrate, people just decided to measure things in kgs.
Outside of canon, I recall an interview with Roddenberry. He put in a speed limit to keep writers from inventing more speed as the way out of trouble. You can see this a few times in the first season where the Enterprise tries to outrun things, but cannot (Qs grid springs to mind).
The TNG Technical Manual also contains a note alluding to Gene's decision.
Figuring out how "fast" various warp
speeds are was pretty complicated, but
not just from a "scientific"
viewpoint. First, we had to satisfy
the general fan expectation that the
new ship was significantly faster than
the original. Second, we had to work
with Gene's recalibration, which put
Warp 10 the absolute top of the
scale. These first two constraints are
fairly simple, but we quickly
discovered that it was easy to make
warp speeds TOO fast. Beyond a certain
speed, we found that the ship would be
able to cross the entire galaxy within
a matter of just a few months. (Having
the ship too fast would make the
galaxy too small a place for the Star
Trek format.) Finally, we had to
provide some loophole for various
powerful aliens like Q, who have a
knack for tossing the ship millions of
light years in the time of a
commercial break. Our solution was to
redraw the warp curve so that the
exponent of the warp factor increases
gradually, then sharply as you
approach Warp 10. At Warp 10, the
exponent (and the speed) would be
infinite, so you could never reach
this value. (Mike used an Excel
spreadsheet to calculate the speeds
and times.) This lets Q and his
friends have fun in the 9.9999+ range,
but also lets our ship travel slowly
enough to keep the galaxy a big place,
and meets the other criteria. (By the
way, we estimate that in "Where No One
Has Gone Before" the Traveler was
probably propelling the Enterprise at
about Warp 9.9999999996. Good thing
they were in the carpool lane.)