Shorter answer: The only objective notion of traveling into the past in general relativity is traveling into one's own past light cone, and if you travel into a black hole, your past light cone does not encompass the the entire infinite future history of the outside universe until you cross a particular boundary inside the black hole which The Science of Interstellar refers to as the "infalling singularity". But the book also mentions that Cooper is scooped up by the Tesseract when he crosses a distinct horizon called the "outflying singularity", and that at this point his past light cone would not include the entire future history of Amelia Brand or the Earth, so as long as he returns to our solar system at a point further in the future than anything included in his past light cone at the moment he was scooped up, he doesn't do any backwards time traveling.
Longer answer: There is no absolute notion of "simultaneity" in relativity--different coordinate systems can disagree on which pairs of events happened "at the same time", so in one coordinate system Cooper's crossing the event horizon might not be simultaneous with any event on an outside observer's worldline (say, Amelia Brand's) no matter how old they get, while in another coordinate system the event of Cooper crossing the horizon could be simultaneous with Amelia Brand's clock showing only a finite time elapsed since he departed from her, although no matter what coordinate system you use, she will never actually see the light from the event of his crossing the horizon ('simultaneity' in a given coordinate system is distinct from visual appearances). For this reason, time dilation, or the rate that one clock is ticking relative to another clock, is coordinate-dependent as well, there isn't really a unique physical answer to the question of how one clock is time-dilated relative to another, although all coordinate systems do agree on how fast each observer can see each other's clock ticking visually if they use light signals, and they also agree on how much time will have elapsed on each observer's clock if they depart from a common point in spacetime and later reunite at another point in spacetime (see the twin paradox).
The only meaningful notion of traveling into one's "own past" in general relativity is when you can enter into the past light cone of some event that you already experienced at an earlier time according to your own clock. The idea of the "past light cone" of a given event A is the complete set of all other events that could have sent a signal, traveling at the speed of light or slower, which would be able to reach A (see here for some basic visualizations of the concept). So at a given event A on your own worldline, any past event you see at that moment using light is, by definition, on the outer surface of your past light cone at the moment you see it, since it's an event which could only influence A by a signal traveling at exactly the speed of light.
So pick the moment right before Cooper gets scooped out of our 3D space and into a higher dimension by the Tesseract. His past light cone at this moment does not include events arbitrarily far in the future of the moment he departed from Amelia Brand--if he aimed his telescope at Amelia Brand it would show only a finite amount of time had passed on her clock since he departed from her, and if at that moment he was receiving a radio signal from Earth (traveling through the wormhole) it would also show only a finite number of years had passed since he had left Earth. The only way his past light cone could include all of the infinite future history of Amelia and the Earth (or an ideal indestructible object alongside each one, since neither Amelia nor the Earth will actually last forever) would be if, as he traveled into the black hole, he saw the entire infinite future history of the universe outside the black hole compressed into a finite period. But as mentioned in this answer from the Usenet Physics FAQ (hosted on the site of physicist John Baez), this doesn't happen when you enter a black hole, at least not immediately after you cross the initial event horizon:
If an external observer sees me slow down asymptotically as I fall, it might seem reasonable that I'd see the universe speed up asymptotically—that I'd see the universe end in a spectacular flash as I went through the horizon. This isn't the case, though. What an external observer sees depends on what light does after I emit it. What I see, however, depends on what light does before it gets to me. And there's no way that light from future events far away can get to me. Faraway events in the arbitrarily distant future never end up on my "past light-cone," the surface made of light rays that get to me at a given time.
This means that if I depart from a friend orbiting outside a black hole when she's celebrating, say, her 30th birthday, both of the following can be true:
She will never see me cross the horizon, i.e. even when she dies at say age 90, the event of my crossing the horizon is still outside her past light cone.
Immediately after I cross the horizon, my past light cone will not include her entire future history--for example, the event of her reaching age 32 might be outside my past light cone, and thus not part of my own causal past.
So, if the rule is just that Cooper can't travel into his past light cone, there's not necessarily any issue with him traveling to a point in spacetime when Amelia was only a few years older than when he left her, even though she never sees him cross the horizon. The exact number of years would presumably depend on some general relativity calculation I don't know how to do, and I'm not sure if the filmmakers actually designed an exact trajectory for Cooper and calculated his past light cone at every point along it up until he was scooped up by the tesseract (this isn't mentioned in The Science of Interstellar). Note however that you can't necessarily assume that just because a brief orbital maneuver near the black hole caused several decades to be added to the outside universe, that Cooper's fall would do the same--gravitational time dilation is not just a function of radius but also of motion, and while an observer hovering or orbiting at a fixed distance above the horizon will see the outside universe aging faster and faster the closer the distance, with the rate of outside aging approaching infinity as the distance above the horizon approaches zero, a falling observer does not see the rate of aging of the outside universe approaching infinity as they approach the horizon (if they did, then their past light cone at the moment they crossed would necessarily include the entire infinite future history of the universe).
Things get even more complex if we consider the case of a rotating black hole, which is what Gargantua in Interstellar is supposed to be. An ideal eternal rotating black hole--a Kerr black hole--would have, in addition to the outer event horizon, a second "inner" event horizon prior to the central singularity, also known as a Cauchy horizon. At this point, there is an infinite blueshift of waves falling in from outside (their wavelength gets compressed to zero), which means that as you cross this boundary you would ideally see the entire infinite future history of the universe pass in a finite time. This is also discussed in the FAQ entry above:
That, at least, is the story for an uncharged, nonrotating black hole. For charged or rotating holes, the story is different. Such holes can contain, in the idealized solutions, "timelike wormholes" which serve as gateways to otherwise disconnected regions—effectively, different universes. Instead of hitting the singularity, I can go through the wormhole. But at the entrance to the wormhole, which acts as a kind of inner event horizon, an infinite speed-up effect actually does occur. If I fall into the wormhole I see the entire history of the universe outside play itself out to the end. Even worse, as the picture speeds up the light gets blueshifted and more energetic, so that as I pass into the wormhole an "infinite blueshift" happens which fries me with hard radiation. There is apparently good reason to believe that the infinite blueshift would imperil the wormhole itself, replacing it with a singularity no less pernicious than the one I've managed to miss. In any case it would render wormhole travel an undertaking of questionable practicality.
And the situation becomes still more complicated if you consider a more realistic rotating black hole. The Kerr black hole is idealized as existing eternally in a perfect vacuum, but the more realistic version would be a rotating black hole that forms collapsing rotating star, and also has light waves and gravitational waves continuing to fall in from the outside even after the original star has collapsed. In this case, the fact that waves falling in from outside become infinitely blueshifted at the inner horizon means that the horizon actually becomes a type of singularity where the energy density goes to infinite, distinct from the singularity at the "center" of the black hole. What's more, Kip Thorne mentions in The Science of Interstellar that recent theoretical work suggests there are actually two distinct blueshift singularities in a rotating black hole distinct from the central singularity, with the more newly-discovered singularity owing to waves that get reflected backwards prior to reaching the inner horizon, and an observer that falls in can cross a boundary where he meets up with all the reflections from waves that fell through the horizon before he did, again infinitely blueshifted so they create a singularity. Thorne labels the singularity at the inner horizon the "infalling singularity", and the newly-discovered singularity caused by reflected waves the "outflying singularity". I quoted from passages where he discussed this in this answer to another Interstellar question, if you're interested.
And Thorne also mentions in The Science of Interstellar that it was decided that the Tesseract scooped Cooper up at the outflying singularity, not the infalling singularity. Thorne also says that observer's past light cone would not include the entire future history of all the waves falling in from the outside universe as they crossed the outflying singularity, unlike with the infalling singularity (the infalling singularity might not include the entire infinite future history of the universe for a realistic black hole since physicists now believe that black holes eventually evaporate, but you would at least be able to see the future of the universe up to the evaporation point as you approached the infalling singularity). Further, it's mentioned that the main reason for this decision was that they had already settled on a science-fictional rule that only gravitational signals could actually travel into a person's own past light cone (which is why Cooper had to signal his daughter using gravitational waves), the Tesseract couldn't actually take Cooper back in time to walk around in his own past or interact in any non-gravitational way with it. From chapter 28 of The Science of Interstellar, p. 249:
When I explained the two singularities to Chris [Nolan], he immediately knew which one should hit the Ranger. The outflying singularity. Why? Because Chris had already adopted, for Interstellar, a variant of the laws of physics that prevents physical objects from ever traveling backward in time (Chapter 30). The infalling singularity is produced by stuff that falls into Gargantua long after Cooper falls in (long after, as measured by the external universe's time; Earth's time). If Cooper is hit by that singularity and survives, the universe's far future will be in his past. He will be so far in our future that, even with the help of the bulk beings, he won't be able to return to the solar system until billions of years after he left, if ever. That would prevent him from ever reuniting with is daughter, Murph.
So Chris firmly chose Cooper to be hit by the outflying singularity, not the infalling one—hit by the singularity arising from stuff that fell into Gargantua before the Ranger, not after it.
And chapter 30 goes into more detail on the rule that only gravitational signals, not people or other objects, can physically interact with their own past (i.e. things within their own past light cone). From p. 263:
Chris made two specific choices for allowed and forbidden time travel—his rule set:
Rule 1: Physical objects and fields with three space dimensions, such as people and light rays, cannot travel backward in time from one location in our brane [i.e. our own spacetime with 3 space dimensions and 1 time dimension] to another, nor can information that they carry. The physical laws or the actual warping of spacetime prevent it. This is true whether the objects are forever lodged in our brane or journey through the bulk [the extra space dimension posited in the movie, which is also possible in some real-world theoretical physics models] in a three-dimensional face of a tesseract, from one point in our brane to another. So, in particular, Cooper can never travel to his own past.
Rule 2: Gravitational forces can carry messages into our brane's past.
So, you can see that they were careful to arrange things so that Cooper being deposited in our solar system less than a century or so after he left was consistent with these rules, in spite of his journey into the black hole.