Well I am no scientist, but for starters let's look at the speeds involved and the likely size and mass of Melancholia.
Firstly, its size and mass. The planet looks from the shots in space to be about 4 times the diameter of the Earth, and therefore if my maths is right, around 64 times the volume of the Earth. Of course we don't know its density, but it certainly looks to be a similar style solid rock planet.
If the density is the same, we have an object around 64 times the mass of the Earth. Ouch.
Next, its speed. John mentions, prior to Melancholia's boomerang move to smash into the Earth, that it is travelling away from the Earth at 60,000 mph. Now, in the cut scene at the start we see the planet, upon impact, consume the entire diameter of the Earth in around 20 seconds. That's around 8000 miles. In 20 seconds!
Even if we charitably assume the collision with Earth did not slow Melancholia down at all, AND don't consider that the Earth is moving away from Melancholia in its own orbit from the Sun, it still meant Melancholia was travelling towards the Earth at around 1.44 million mph at the point of impact. That's around 1/465th of LIGHT SPEED (186,000 mps * 60 secs * 60 minutes / 1.44 million mph = 465).
Please someone correct if my maths is way off, but the numbers look approximately correct to me!
What kind of gravitational force is required to make something considerably more massive than the Earth (64 times is my ballpark guess), travelling at 60,000 mph away from it and getting it to travel at 1.5 million mph towards it? My humble guess is that it would take a lot more than the Earth's puny 1G. A very close fly-by would probably hardly bend its trajectory at all with Melacholia being so massive by comparison.
An exo-planet being tossed out of its solar orbit and eventually finding our solar system is exceptionally improbable, but certainly not impossible. But I think the science of its orbit could have been worked better. Maybe initially passing the Earth and moving away, and then some months later meeting the Earth again after a trip past the Sun (something as massive as the Sun could certainly bend the planet's trajectory if it got close enough, as it was supposed to have done in the movie before reaching Earth).
Another couple of scientific flaws I have noticed:
1) In the scene where they are on the balcony at night viewing Melancholia's moon-style rise, we hear the planet rumbling away. How would that sound travel through the total vacuum of space to the ears of the observer? We'd hear nothing, because there would be nothing for it to travel through.
2) Does anyone else but me think the planet is a bit too beautiful and blue and swirly considering it's been travelling through space at temperatures of near absolute zero for probably tens of thousands of years at least, then gets cooked by our Sun on its way past? :)
3) Wondering if tidal forces would have ripped the planet and final scene apart long before impact given the mass difference as things started to "fall" towards the massive planet at 64G?