In the film The Avengers, the main characters becoming a team relies on the following:
Iron Man: If Stark had gone in the Humvee with Rhodey, he would never have been captured and never would have become Iron Man.
Captain America: If he wasn't found in the ice before the events of the Avengers film, he would never been part of the team.
Thor: If Thor had not learnt self sacrifice, or had he been punished and sent to a different world, he would never have been part of the team either.
If these main characters had never been our defence against the Chitauri invasion in the Avengers film, we would have been significantly worse off. Thanos would have still sent the invasion to claim the Tesseract (although they might have a leader other than Loki). We would still have the Hulk, Black Widow and Hawkeye on our side, but I doubt on their own they could have fought off the Chitauri invasion.
The Ancient One and the Masters of the Mystic Arts have obviously been around for a long time. During the film Doctor Strange the Ancient One mentions that the "Avengers tackle physical threats, whereas they dealt with the mystical threats" (I'm paraphrasing here).
However if that Avengers had never come together to save the world from the Chitauri invasion, would the Ancient One and the Masters of the Mystic Arts have intervened at all? Would they let the Earth fall under the control of Thanos?
Would their magic be sufficient to push the Chitauri back?
2022 EDIT:
Having now seen Avengers Endgame, it does indeed show the Ancient One defending Earth from the Chitauri invasions during the events of Avengers 1 - so I guess the answer to my question is: yes!