The basic premise of the rebooted Start Trek films is that Spock and Nero traveling to the past caused an alternate time line. But there have been other instances of time travel in the Star Trek universe which don't seem to create any alternate realties (The Voyage Home, First Contact,...). Is there some sort of reason for this, or are the producers just very inconsistent?
This was (kinda) explained in the Voyager episode "Relativity". In short, time travel does indeed result in a wide range of paradoxes, some relatively benign, others dramatically catastrophic. It's only those events that are not pre-destined and have a large effect on other planets and civilisations that seem to result in the creation of a new timeline. Those events that are pre-determined (or just pretty minor, and hence otherwise unimportant in the grand scheme of things) don't.
The key determiner appears to be scope. The bigger the effect and the wider the physical area affected, the greater the prospect of a time-travel event ripping the universe a new one.
DUCANE: Let's see how much you've assimilated. The Dali paradox.
SEVEN: Also known as the Melting Clock Effect. It refers to a temporal fissure which slows the passage of time to a gradual halt.
DUCANE: The Pogo Paradox.
SEVEN: A causality loop in which interference to prevent an event actually triggers the same event.
DUCANE: Excellent. Can you give me an example?
SEVEN: The Borg once travelled back in time to stop Zefram Cochrane from breaking the warp barrier. They succeeded, but that in turn led the starship Enterprise to intervene. They assisted Cochrane with the flight the Borg was trying to prevent. Causal loop complete.
Time travel is rarely, if ever, handled consistently in any sci-fi series. That said, most of the time either A) a new timeline is created, B) the characters (either in the new timeline, or crossing over from the old one) "fix" whatever changed so that history rights itself, or C) some combination of A and B.
Star Trek: The Original Series
"Tomorrow is Yesterday": The Enterprise goes back to the 1960s and accidentally kidnap a fighter pilot, Captain John Chrisopher. At first they think it's no big deal, because Christopher wasn't going to contribute significantly to history. But then Spock realizes that his son would, and his son hadn't yet been born. So they fix what they did wrong, preventing an alternate timeline from forming.
"The City on the Edge of Forever": McCoy goes to the past, and creates an alternate timeline in which the Federation never existed. Timeline remnants Kirk and Spock follow him to the past, prevent him from changing history, and so history is righted and the alternate timeline is wiped out.
Star Trek: The Next Generation
"Yesterday's Enterprise": The Enterprise-C travels to the future, leaving its proper moment in history with a significant job left undone. This creates an alternate timeline in which the Federation is at war with the Klingons. They manage to send the Enterprise-C back, theoretically righting the timeline. But it turns out that starting from this episode onward, they were in fact living in an alternate universe - one in which a Romulan had conceived a child with a human prisoner named Tasha Yar, and that child, Sela, grew up in the Romulan Empire despising humans and the Federation.
"Time's Arrow": Here, we have a stable time loop. Data's head was found in 24th century Earth because at a later point, he went back to the past. His head became separated from his body and stayed there, while his body traveled back to the future to rejoin the head they'd already found. Stable loop, no new timeline created - it was the same timeline that had always existed.
Star Trek: First Contact: the Borg go back in time to destroy Earth in the past. This creates a new timeline in which the Borg have taken over the Alpha Quadrant. Timeline remnants Enterprise-E follow them back and prevent their takeover, thus wiping out the alternate timeline.
"Firstborn": A future Alexander Rozhenko travels to his past (the series' present) specifically to create an alternate timeline. He witnessed his father dying, and attributed it to his younger self's refusal to become a warrior. As he lay dying in his father's arms, Worf tells him that the changes he's made to the past may in fact have already altered the timeline. We never saw the future, so for all we know, from that moment on, the crew is in an alternate universe, one in which Worf will not die the way Alexander saw.
Star Trek: Deep Space Nine
"Past Tense": Sisko, Bashir, and Dax travel to the past, and almost immediately screw things up, by causing an incident that results in the killing of a man named Gabriel Bell. The rest of the Defiant crew now find themselves in an alternate universe, where the Federation does not exist. Sisko then plays the part of Bell, restoring history... almost. As with "Yesterday's Enterprise", the original universe is restored, but not quite. In this new universe, official Starfleet records show that Gabriel Bell actually had the appearance of Ben Sisko (apparently the only known photograph of him was taken after Sisko assumed his identity?)
"The Visitor": Much the same as "First Born", this time it is Sisko's son who travels to the past to prevent his father dying. He does so successfully, thus wiping out his original timeline, in favor of an alternate universe created in which Sisko was not lost in some temporal phenomenon.
This regular pattern is not upheld in the movie Star Trek. What changed in the past this time was the destruction of the USS Kelvin, and later of the planet Vulcan. Only one person from the original timeline had managed to follow the bad guys back to the past, and he was unable to prevent either the Kelvin's or Vulcan's destruction. Thus the timeline remained altered, and did not right itself as it had so often.