During the episode "Simon & Marcy":
>!Marcy mentions that Simon's features have changed each time he takes the crown off. Simon is also very reluctant throughout the episode to put the Ice Crown on, because each time it becomes harder and harder for him take off again. Each time he puts it on, he keeps a little piece of the Ice King's features in his personality and in the way he looks. The last time he puts the Ice Crown on, he sings a song that reminds him of his previous self. This is the only way he is able to come back to help save Marcy (with the help of the goo). If you notice, when he takes it off the final time in this episode, he barely resembles Simon at all anymore and looks almost completely like the Ice King.