In the film, it is shown that there are many alien species coexisting in the galaxy. Clearly, they know where Earth is and they know about humans, since Starlord was abducted from there, and they speak of "Terrans" multiple times. But how come the interactions with Earth are so limited?
The short answer: The Galactic community does not generally interact with Earth because Earth is not the center of its own galactic empire. It does not have a unified planetary government. It is a civilization still, despite the existence of metahumans and the occasional burst of advanced technology, isolated to one planet in its solar system.
In comparison to the planets we have seen in the galactic community, Earth is home to a low-tech civilization with no appreciable industries, racial and social stratification, rampant violence and a dangerous metahuman population.
Marvel Cinematic Universe
In the Marvel Cinematic Universe, we have been introduced to a tiny sliver of the galactic community that includes the planet Xandar, home of the Nova Corps and the Kree Empire. (The Kree, in the canon Marvel Universe, live in the Large Magellanic Cloud, essentially another galaxy near the Milky Way.)
We take a quick stop at Knowwhere, a Celestial anomaly at the edge of known space, home of miners and reprobates, and a number of smaller planets where the Ravagers hide and former civilizations once thrived. This is not a grand demographic of the galactic community in the Marvel Universe. If the Marvel Universe is anything like our own, there are likely hundreds of millions of potential planets littering their Milky Way galaxy alone.
To the galactic community Earth is relatively insignificant except for its connection to the Nine Realms (Asgard and associated systems). Earth is a planet with no unusual resources, no space fleets, and no extraordinary cultural resources. In other words, no reason for aliens with interstellar or intergalactic travel capabilities to visit unless they want to see a primitive world.
Judging from the fact that many of the Xandarians resemble Humans/Terrans physically, and evidently at least one Human (Star Lord) has been seen/arrested in galactic space, our existence is known to them. But since Earth has no interstellar or intergalactic flight capacity we don't know they exist unless they visit us.
Since Star Lord's father was Human enough to mate successfully on Earth, the difference between Humans and Spartans must be minimal. They have seen our species before and we are not all that interesting to them.
If I were to make a comparison, we would be more likely to enjoy ourselves more if we were to visit the Amish, with their aversion to some aspects of technology than most galactics would have visiting Earth with what they would consider to be a barely civilized world without any of the creature comforts they would be used to.
Visiting Earth might be the equivalent of camping with no connection to advanced technologies, information databases, and exotic galactic cuisine and any other things we have no idea could exist in an advanced technological society. Visiting Earth and maintaining a low profile could be the equivalent of "roughing it."
Well, tell that to the folks in Manhattan. The Chitauri interacted rather heavily with their island.
And as Thor said in "The Avengers", only the Tesseract research told the other realms that planet Earth had reached a certain level of advancement that makes it interesting enough to be considered as not being too primitive.
Then again, maybe they have interacted before, but it was all covered up: As Phil asks Sky in "AoS - 1x02 - 0-8-4":
"Remember the panic when that Anti-Matter meteor splashed down just off the coast of Miami and nearly devoured the city?"
Earth is a planet with continuous developments in technology, culture, etc. In the latest Guardians of the Galaxy comics, J-Son of Spartax (aka. Star-Lord's dad) treats Earth as an isolated planet but fears of its potential. That's why during an another alien attack on Earth (which was defended by the Guardians), J-Son ordered his men to arrest the Guardians due to disobeying his order not to interfere with Earth stuff, even though they just defended Earth's safety. On the other hand, MCU's Thanos wants to invade Earth because there may be someone ('dem superheroes) who could challenge his power (or maybe it's his fascination with death, idk). There's one thing I had in mind: after the Chitauri invasion in New York, would it widen ordinary people's minds about extraterrestrial life? Would there be cults or further research?