Harry survived his encounter with Voldemort as a 1-year-old because of his mother's sacrifice, not because of the powers Voldemort transferred to him (which is what Dumbledore is talking about in the quote).
"Your mother died to save you. If there is one thing Voldemort cannot understand, it is love. He didn't realise that love as powerful as your mother's for you leaves its own mark. Not a scar, no visible sign... to have been loved so deeply, even though the person who loved us is gone, will give us some protection forever."
—Dumbledore regarding the power of love, Harry Potter and the Philosopher's Stone, Chapter 17, The Man With Two Faces
"[Voldemort] gave you powers, ..., which have fitted you to escape him not once, but four times so far — something that neither your parents, nor Neville’s parents, ever achieved.”
—Dumbledore explaining Voldemort's transferral of power into Harry, Harry Potter and the Order of the Phoenix, Chapter 37, The Lost Prophecy
So, Harry survived 5 out 5 encounters with Voldemort since birth, at the time of Order of the Phoenix, 1 of which (the first) was thanks to his mother's sacrificial love magic, and 4 of which were thanks to the powers Voldemort transferred (as Dumbledore describes in your quote in question).
Consider that Harry didn't have Voldemort's powers when he survived his attack as a 1yo (it was his mother's sacrifice that did it). Ergo, Voldemort's attempt when Harry was a 1-year-old doesn't count. So, Dumbledore's count of four times is correct. J.K Rowling isn't so bad at maths after all.