I seem to recall personal communicators having various troubles such as being out of range in TOS. Was the distance Kirk called Scotty from in Star Trek Into Darkness comparable to the range of the communicators in the original series?
Not even close. Into Darkness seems to have a range and speed of communication, not even considered in the Prime Universe.
In the Prime Universe's Star Trek, the Original Series, the limit to a communicator was from the surface of a planet to the ship in orbit above it. Ships could communicate between locations but there was some lag, from minutes to days depending on the distance from Earth to the starship.
Even in the time of Prime DS9 or Voyager, even the fastest communications from the Delta Quadrant (30,000-40,000 light years) had a relay time of 11 minutes each way. This was the fastest technology possible in the Prime Universe for the Federation. I don't ever remember a real-time communication from Earth to anyplace at the edge of Federation territory.
Strangely enough, Kirk is able to, from the EDGE of the Klingon Neutral Zone to reach Montgomery Scott who is either on Earth or in the Sol System somewhere. Assuming he is relaying this through the ship's communications array, the Klingon Neutral Zone is very close to Earth or the speed of Star Fleet communications has improved tremendously. There is insufficient information available to determine what has improved communication technology so radically from the Prime Star Trek Universe.
The ability to send data across "sub-space", which allows for far-FTL communication, is a key future-tech that has always been part of Star Trek. In many examples, it's apparent that sub-space comms can stretch across the galaxy with minimal delay, provided the proper equipment is available. We see, in many episodes especially in DS9, the ability for characters to hold real-time conversations with people on the other side of the Federation (DS9 to Earth, for instance, which is according to this map about the same distance as Earth is to the Klingon border in line with Qo'nos).
There are, in fact, two instances in the original TOS-era movies where real-time communication across what's probably a long way are demonstrated. First, in Wrath of Khan, Kirk and Dr. Carol Marcus have a somewhat successful real-time conversation between the Enterprise in the general vicinity of Earth, and Regula I out in the hinterlands, even though it's being jammed by Khan's crew aboard the hijacked Reliant. This is, according to the map, the longest distance over which any real-time communication was shown; about 1.5x the next longest distance (from Earth to Qu'onos), and dwarfing the communication distance from Earth to the Klingon border in Into Darkness. Then, in The Undiscovered Country, we see what appears to be real-time communication between Qo'nos and Earth for Kirk and McCoy's trial, but this didn't have to be two-way (so it may not in fact have been real-time) and it was broadcast pretty much everywhere in the known galaxy. Other communication in these two movies and others, including between Enterprise near Rura Penthe and wherever Sulu was (which was probably also near the Neutral Zone) isn't as impressive as other examples.
It's conceivable that the arrival of the Narada, and Spock, into the alternate timeline has also heralded the arrival of some TNG-era technology, for example trans-warp matter transportation, which may also have brought with it new paradigms in sub-space communication. It's also no small amount of hand-waving, even given the requisite suspension of disbelief for FTL communication in the first place. It seems to be a given that Kirk's communicator was being routed through the Enterprise's comms array, and that somehow this allowed Scotty's comm signal to make the return trip. However, we are given no explanation other than the obvious; this ain't your daddy's Star Trek.