Why was Nimoy in the 2009 Star Trek movie and Shatner not?

Question

The story line of Star Trek (2009) explains that Spock was sent back trying to help Romulus but failed so he was forced to watch Vulcan get destroyed.

But, out-of-universe, why was Shatner not asked to appear in the movie?

Answer 1

In the simplest terms, Nimoy was happy to accept a small cameo role whereas Shatner wasn't.

J.J. Abrams specifically addressed this issue (both in and out of universe) in an interview with Sci-Fi Scanner;

“It was very tricky. We actually had written a scene with him in it that was a flashback kind of thing, but the truth is, it didn’t quite feel right. The bigger thing was that he was very vocal that he didn’t want to do a cameo. We tried desperately to put him in the movie, but he was making it very clear that he wanted the movie to focus on him significantly, which, frankly, he deserves.

The truth is, the story that we were telling required a certain adherence to the Trek canon and consistency of storytelling. It’s funny — a lot of the people who were proclaiming that he must be in this movie were the same people saying it must adhere to canon. Well, his character died on screen. Maybe a smarter group of filmmakers could have figured out how to resolve that.”

If it makes you happy, there were apparently several attempts to write him in but the writers claim that there simply wasn't a good enough reason to skew the story purely in order to allow a two minute continuity-porn cameo:

The Shatner ending of “Star Trek” was abandoned for a whole variety of reasons. “Whereas our elder Spock had a very organic reason to be there, we didn’t have that same benefit with Kirk,” Kurtzman explained. “Because Kirk died in the movies — he died in canon — it was very hard to come up with a way to bring him back in the movie that didn’t feel contrived.”

“Ultimately, we decided internally that we were split,” Orci remembered of the decision to abandon the Shatner ending. “The decision was that it wasn’t quite enough to justify wasting his time.”

Still, it’s pretty obvious where Orci fell in the internal debate. “It was a nice voice-over. It was more than a scene,” he explained. “I think it could have worked, personally.”

---

William Shatner referenced this "cameo" in an interview on his website. According to him, he was never asked to be in the film but I suspect we can take this with a grain of salt...

Answer 2

Romulus was destroyed in 2387, Kirk dies in 2371 (actually, even 78 years earlier, if we don't take Nexus in account). It would be really hard to explain his appearance.