In the Potterverse, a wizard's wand normally stays loyal to its original owner; the wand chooses the wizard. This is seen throughout the books/films, but shown explicitly in book 7 when Harry's new wand (given to him by Ron after he took it from one of the Snatchers) fails to perform as his old holly wand did, and when Hermione, as Bellatrix, doesn't feel right holding Bellatrix's wand. In both cases, the person holding it did not win the wand personally from its previous owner. On the other hand, Draco's wand behaves much better for Harry because he physically disarmed Draco to get it.
Most wands will retain some allegiance to their former owners; should that owner reclaim the wand, even without "winning" it from the current owner, it's implied that the wand will accept their old owner. Also, in most cases of disarmament (Expelliarmus), the wand is usually given willingly back to the person who was disarmed, whether the disarmament was for practice or for real. So, the wand either wouldn't have much chance to change allegiance, or in being handed back willingly the allegiance would change right back.
There is one case of a wand working for someone who didn't properly win it; Tom Riddle's memory-self using Harry's wand in Chamber of Secrets. He doesn't do much with it, true, but in the book universe a wand would be less powerful or not work at all for a user other than its owner, while in the films a wand that hadn't chosen its current handler would be uncontrollable. This may be explained by the "twin cores"; the same phoenix (Fawkes, in fact) gave two feathers, which ended up in Voldemort's and Harry's wands. This may make both Voldemort and Harry more able to use each others' wands (though this theory isn't tested anywhere else in the books or movies).
However, the Elder Wand is different. It will give its allegiance only to one wizard, the one who "won" it from its last owner. It responds, as was said, only to power. In the overwhelming majority of cases, it changes hands in a very bloody fashion; the old owner of the wand isn't going to come looking for it. Only in the last 70 years or so preceding the events of the books did it begin changing hands while the previous owner remained alive. Grindelwald stole it from Gregorovitch sometime prior to 1945, and Gregorovitch lived for decades thereafter until Voldemort killed him. Likewise, Grindelwald was defeated by Dumbledore in 1945 and spent the rest of his life in Nurmengard Prison before Voldemort killed him too. In Book 6 Dumbledore then lost the wand to Draco, who didn't kill him. Then Draco, who never touched the wand, lost ownership when Harry bested him at Malfoy Manor in Book 7.
The point about a wand's allegiance changing was indeed paid more attention to in the last book, because it was central to the plot of that book. However, the knowledge that "the wand chooses the wizard" was known to readers and moviegoers since The Sorcerer's Stone, when Harry bought his holly wand; Ollivander said as much to Harry in his shop.