The change in Neville is slow, but it starts in Book 1.
Neville's biggest problem has always been that his confidence (or lack thereof) effected his ability to perform. But starting from the very beginning, events have conspired to boost Neville's confidence.
In Book 1, Dumbledore affirms Neville's decision to confront the trio, calls him brave, and makes him the hero of Gryffindor.
In Book 3, Lupin and the boggart help him face and symbolically defeat his fears. He's still afraid of course, but conquering your fears once makes it easier to do it again.
In Book 4, Mad-Eye Moody, possibly the greatest Auror of a generation, singles out Neville and affirms his skills in Herbology. This was part of a plan to help Harry, but there's no evidence that Neville ever learns of that. Even if he does, he doesn't learn until much, much later. Also, he successfully asks a girl out, which is a rather big deal to a 14-year-old.
In Book 5, Neville studies Defense Against the Dark Arts under Harry. As someone whose primary limitation is his confidence, learning under a peer is good for him, and his skills grow rapidly. He then proceeds to accompany Harry in their raid on the Department of Mysteries and faces off with Voldemort's Death Eaters, including Belletrix who tortured his parents, and comes out okay (well, alive and unharmed, at least).
Speaking of his parents, Neville likely regarded his parents' condition with a mixture of pride and shame. Pride because they were injured in the line of duty, and shame because infirmity (particularly mental infirmity) is stigmatized. But in Book 5 Neville's friends learn about his parents, and they proceed to not treat him any differently. It's a small thing, but it likely matters.
Also, Neville's second-hand wand breaks in the Department of Mysteries, and is replaced by a new wand. New wands are generally superior to hand-me-downs.
All of this combined (and more) is what makes Neville the leader of the Hogwarts rebellion, and gives him the confidence to be a true Gryffindor, summon the Sword, and kill Nagini.