In the opening scenes of Melancholia, I was not totally sure whether the larger planet was Earth or Melancholia. Is the size of Melancholia ever mentioned? Or can it be answered from a valid source elsewhere?
As stated in this interview on the movie's official webpage, the planet known as Melancholia was envisioned to be "ten times bigger" than Earth:
"Let us get it over with right away. The end of Lars von Trier's film 'Melancholia'. Everybody dies. Not just the guests at the grand wedding held in the first part of the film at an ever-so-romantic castle surrounded by a golf course. And not just all life on Earth. For in the world evoked by the Danish film maker this time, we are absolutely alone in the universe. So what ends in our planet's cosmic embrace with the ten times bigger planet, Melancholia, is life as such and our recollection of it."
Since this was an art film about the varied human responses to the end of the world, less physics was applied than would be desired by the scientifically-inclined among us. Melancholia was considered to be a massive telluric superplanet, making it at least as large as Earth.
Judging from the graphics used in the movie, it is hard to believe the planet would be more massive than Earth seeing how it swings around the Earth or at least that's what their computer graphic implies. This clip shows the planet as it slingshots around the sun, at that point it should be moving very fast and unable to swing back toward the Earth after passing it.
Yes, there may be gravitational fluctuations as the two bodies pass each other (maybe even destabilizing the Earth's and the moon's mutual orbit), but there should have been no way for the Earth to have cause Melancholia to swing back toward the Earth, particularly if it was as massive as the Earth.
Melancholia, a massive blue telluric planet (or super-earth) that had formerly been hidden behind the sun, becomes visible in the sky, approaching Earth. John is excited about the planet, and looks forward to the "fly-by" expected by scientists, who have assured the public that Earth and Melancholia will pass each other without colliding.