Harry survived his encounter with Voldemort as a 1-year-old because of his mother's sacrifice, not because of the powers Voldemort transferred to him (which is what Dumbledore is talking about in the quote).
"Your mother died to save you. If there is one thing Voldemort cannot understand, it is love. He didn't realise that love as powerful as your mother's for you leaves its own mark. Not a scar, no visible sign... to have been loved so deeply, even though the person who loved us is gone, will give us some protection forever."
—Dumbledore regarding the power of love, Harry Potter and the Philosopher's Stone, Chapter 17, The Man With Two Faces
"[Voldemort] gave you powers, ..., which have
fitted you to escape him not once, but four times so far — something
that neither your parents, nor Neville’s parents, ever achieved.”
—Dumbledore explaining Voldemort's transferral of power into Harry, Harry Potter and the Order of the Phoenix, Chapter 37, The Lost Prophecy
So, Harry survived 5 out 5 encounters with Voldemort since birth, at the time of Order of the Phoenix, 1 of which (the first) was thanks to his mother's sacrificial love magic, and 4 of which were thanks to the powers Voldemort transferred (as Dumbledore describes in your quote in question).
Consider that Harry didn't have Voldemort's powers when he survived his attack as a 1yo (it was his mother's sacrifice that did it). Ergo, Voldemort's attempt when Harry was a 1-year-old doesn't count. So, Dumbledore's count of four times is correct. J.K Rowling isn't so bad at maths after all.
Conclusion:
But, if you count the last encounter between Harry and Voldemort, at the ministry, it's actually f̶i̶v̶e̶ ̶t̶i̶m̶e̶s̶,̶ ̶n̶o̶t̶ four.
- V̶o̶l̶d̶e̶m̶o̶r̶t̶'̶s̶ ̶a̶t̶t̶e̶m̶p̶t̶ ̶w̶h̶e̶n̶ ̶H̶a̶r̶r̶y̶ ̶w̶a̶s̶ ̶1̶ ̶y̶e̶a̶r̶ ̶o̶l̶d̶
- In the first book, when they fight over the stone
- In the second book, when Harry meets young Tom Riddle
- In the fourth book, when Harry joins V's rebirth party
- The fight at the ministry, in the previous chapter
Note: It would be reasonable for readers/audience, out of universe, to assume that Riddle's Diary, being a horcrux, is not the true Voldemort, and thusly it doesn't count as one of the four, but keep in mind that, in universe, Dumbledore, certainly during this conversation with Harry in Order of the Phoenix, doesn't have a particular technical discernment between true Voldemort and "non-true"/Horcrux Voldemort, especially when talking to his student, Harry, at this point in the story. Moreover, there is evidence in this screenshot of the Chamber of Secrets book that confirms that Dumbledore refers to The Diary, or the culprit of that year's events, as Voldemort, meaning that it stands to reason that during his Order of the Phoenix conversation with Harry, he would include Harry's encounter with The Diary as one of the four escapes.