Off the top of my head, Azkaban, apart from being the emotional "hinge" in the series that shifts it from child to adult themes, introduces -
- Sirius Black. Crucial to later books.
- Remus Lupin. Crucial to later
- Revealing Scabbers to be Peter Pettigrew, whose escape triggers
literally the next 4 books' worth of plot.
- The Marauders plotline.
- The Marauders' Map.
- The whole Fidelius Charm plotline. Crucial to backstory.
- Professor Trelawney and the notion of Prophecy in the HP universe. Crucial to later books and backstory.
- Hagrid's new job, also important for the next books.
- Dementors and Boggarts, reflecting the themes of depression and
fear, respectively. Crucial to later books.
- Hogsmeade and the Shrieking Shack. Crucial to later books.
- Werewolves and the notion of segregation.
- Griffindor finally winning the Quidditch cup. Big deal when
reading them in real time :P
- The Nimbus 2000 being destroyed.
- The Firebolt.
These are just the examples that are referenced a bunch in later books or carry some sort of important story theme, like character development (the Nimbus 2000 being destroyed is a big deal to 13-year-old Harry). This list ignores all the intangible but crucial stuff, such as normal character development (for example, Harry's first romantic thought in the series occurs in a Quidditch match in book 3).
You also have to remember that the movies boil down (coughbutchercough) and simplify the plots of the books massively (especially past 2), so the importance of Azkaban is a bit tricky to see from a movie point of view. Many consider it their favourite HP book, and I'd recommend it, especially since it's a definite change in gear compared to book 1 and 2.
By the way, book 6 has no Harry vs. Voldemort moment either ;)