There's no reference in the books, but Sanderson is using real-world metallurgy by implication.
Aluminum is rare before society reaches industrial levels of tech, because it requires more than basic smelting and shaping to have in sufficient quantities to make cheap shiny hats.
To compare to the real-world (and shamelessly steal from Wikipedia), aluminum as an element is the third most common material in the Earth. The problem is that as a pure metal in the ground, it reacts and bonds with so many other things that you soon have almost no pure metal - it becomes a bunch of other minerals like silicates. The easiest way to get it out of the ground is through a mineral called bauxite (as Valorum notes in comment) and it requires (I'm skimming here) significant temperature, pressure, and a rotary kiln, the first of which a medieval setting might provide, but the rest highly unlikely. Once you have it, it bonds with air just as quickly, but in a way that prevents corrosion to the larger portion of the metal.
So whatever aluminum the Final Empire had - and only the Lord Ruler and his Inquisitors seemed to have any (I presume Vin's metallurgist got it from her, and she got it inside the abandoned Kredik Shaw) - they got from a rare natural deposit, possibly experiments on ores, or a stash left over from the times pre-Ascension whose tech level is poorly understood.
(it is also entirely possible, since the nature of the world was changed at the end of the trilogy, that it wasn't "immune/inert" at first, but there is no way to prove that with the info we have, and Sanderson using metal history as a template is much more likely)