In the Adventure Time season five episode, "The Party's Over, Isla de Señorita," the Ice King spends a significant portion of the episode powerless because he isn't wearing his crown. This is expected and consistent with established canon.
However, in the earlier episode "Simon & Marcy," we see:
When Simon removes the Ice Crown, he reverts to a milder version of himself, the hippie version who cares about Marceline and remembers the Time Before the Mushroom War. We know (thanks to that same episode) that the power of the crown increases dramatically each time Simon uses it.
However, why doesn't removing the crown do this anymore?