You don't understand Katara. The only way to get to the Airbender temple is on Flying bison, and I doubt the Fire nation has any flying bison
And yet the Air Nomads were killed by the Fire Nation in 0 AG
However, its population was wiped out during the Air Nomad Genocide in 0 AG
Source: Eastern Air Temple
If I recall Aang's comment to Katara was his hopeful optimism that there had to be Air Benders left. Because he though that the only way into the Eastern Air Temple was by flying bison then the Air Nomads at the Eastern Air Temple, at the very least, were safe.
So how did Guru Pathik get there? We don't know for sure but we know the place is scaleable to non-Air Benders as shown in The Legend of Korra Book 2 - Spirits when Tenzen, Kya and Bumi are looking for Ikki after she runs away. While Tenzen is an Air Bender and Kya a Water Bender Bumi isn't a Bender and the 3 were going down the paths outside the main temple grounds long into the night as Ikki was found the following morning in a cave so it would suggest they were going down for a good while.
Also despite it being Aang's home neither he nor Tenzen knew about the Meditation circle:
The meditation circle is located some distance from the main temple. The area was not known by the Air Acolytes, and due to its neglect, there was an overgrowth of vegetation and reduced spiritual connections.
And in the Trivia's 3rd point:
Along with the other three ancient air temples, the Eastern Air Temple was restored to its former glory by Avatar Aang
Yet as quoted above the circle was neglected, needed cleansing and it took Jinora leading the group (who was being led by dragonfly bunny spirits) to find it as it would have been Tenzen's first choice of mediation spot when he was training Korra.
Also finally we see in a flashback the Eastern Air Temple in flames:
And while not entirely clear I would say that the positions of where the fire is coming out from and the lack of smaller fires further down would suggest more that the Fire Nation got into the temple themselves.
Also as Ellesedil as pointed out in the comments, there is also the Northern Air Temple which during the events of The Last Airbender Book 1 Episode 17 "The Northern Air Temple" where a group of Non-Benders (particularly Earth Kingdom Refugees) who, without flying bison, was able to take up residence and even got into the Air temple sanctuary which was supposed to only be unlockable by a master Air Bender.
During the episode we also see the Fire Nation attacks the temple with a combination of foot soldiers and Tundra tanks1 and we see that the foot soldiers don't have too much of a problem climbing the slopes as such, just as Ellesedil said "Presumably, climbing is always an option"
This shows us that Aang's comment was just optimism as he doesn't know everything about the Western Air Temple and what happened there.
Also given the amount of time that passed between Aang first revisiting the temple in Book 1 to Appa appearing their after he was separated having been kidnapped by the Sand Benders, it's not out of the realm of possibility that Guru Pathik found some out of the way path that allowed him to scale the mountains into the temple, is not the same path used by the Fire Nation when they killed the Air Benders, if not having climbed himself.
1: The wikia page says the tanks were created in the late stages of the Hundred Year War. Since it was Air Nomad Genocide which actually sparked the War (it was this event which lead the Water Tribes and Earth Kingdom to declare war against the Fire Nation) we can assume that these tanks were never used to siege the Temples.