The Observers' invasion would have happened either way, right? So all the events from season 1 through 4 do not actually prevent or affect that, right?
Yes, sort-of. It's never explicitly stated, but the implication is that, had none of the events of Fringe happened, the Observers would have originally invaded the alternate universe. What we see during the first three seasons was the Observers adapting their plan to try and ensure that the invasion could happen, after September put it in jeopardy.
If we piece together the things we learn about the Observers (largely from September's brain dump to Peter), we can figure out that the Observers future is the original future from the alternate universe. Once they realized that world was dead, they picked a team of scientists, the ones we call Observers, to go back through time and observe humanity, and select a time in history when the invasion should happen. This is the original reason they showed up at so many crucial events.
Unfortunately, during one of those events, September accidentally "destroyed" his own future when he caused alternate-universe Walter to miss finding a cure for Peter. Their attempts to fix it (using prime Walter as a "tool") backfire, kicking off the war between universes, and driving the plot of the first three seasons. Eventually, they were able to get Peter to use the device to correct their mistakes, and allow him to disappear from history (September went back and allowed Peter to die the night Walter kidnapped him).
In Season Four, things get a bit fuzzy. The show does not ever really explain why Peter was allowed to die, when he was originally deemed important enough to save. It also never explains why his subsequent return to the wrong universe, and hooking up with Olivia (who is from different universes) and having a child, are all allowed to happen without screwing up the Observer's future. In fact, it's never really made clear why the prime timeline is of any importance to the Observers (which is clearly still is), if they're from the alternate universe. However, the Observer's technology gives them pretty broad leeway to observer and manipulate and travel through time, so the show mostly just handwaves those questions away as "the Observers know what they're doing."
After all of the Observers manipulations played out, the prime timeline was now deemed suitable for their invasion instead of the alternate one, in the year 2015.