In Ragnarok, Thor says Sakaar is the second alien planet Banner had been on. What was the first?
The actual quote is
Banner: Calm? I'm on an alien planet
Thor: It's just a planet. You've been on a planet before
Banner: Yeah, one.
Thor: Well, now you've been on two.
So the clear implication is that the one planet that he's already been on is Earth, with Sakaar being only his second.