In the Mythic Era of "The Elder Scrolls" lore, Nirn was populated solely by the ancestors of the beast-races (Argonians, Khajiit, Sload, etc.) It is said that fragments of another world - Ehlnofey - crashed on Nirn, bringing with it the peoples of that planet (who were also referred to as "the Ehlnofey"). Basically, aliens landed on Nirn.
These "aliens" split into two factions, one eventually becoming the bloodlines of Men and the other becoming the bloodlines of Mer. The faction that would evolve into the Mer had part of their city intact, and therefore had access to technology and architecture the peoples of Nirn had never seen before. This is likely the basis of Aldmeri domination over "magic".
Are there any indications in the games or books that Ehlnofey might have been Earth, or an Earth colony/ship?
I know this is probably just fishing, but it would explain why Elves and Men are genetically compatible (they can have children together, after all). It would also explain how items from Nirn can land on the earth of Fallout or vice versa - they're in the same galaxy!