You're looking for the animated films featuring "Unico" :-)
Unico is a baby unicorn with white fur, a pink mane, and little
cinnamon bun-shaped ears, who was born with the very special gift of
making all living creatures lighthearted and happy. His friends
include Beezle, the young Devil of Solitude; Chao ("Katy"), a naive
little kitty who longs to be a human girl, and to learn magic from a
real witch; a spunky little Sphinx, and a warm-hearted human girl
named Cheri.
The Fantastic Adventures of Unico (1981)
This is the film where he loses or rather gives away his horn and gets into big trouble....
[Unico] agrees to hand over his horn to Beezle if he promises to be
Best Friends Forever. With Unico's horn now part of him Beezle becomes
even more 'spirited' (unbearable) and chases Unico around the island
and eventually off a cliff, into the sea. Without his horn Unico is
powerless and begins to drown...
Unico in the Island of Magic (1983)
This is the one with the castle made up of human building blocks....
Lord Kuruku plans to turn all living creatures, animals and people
alike, into zombie-like beings called "Living Puppets" [these are the
scary human building blocks in the movie] to be his slaves. Toby
[Cheri's brother] works for Lord Kuruku, his job is to change people
into Living Puppets and then lure them to Kuruku's island castle in
exchange for learning more of Kuruku's magic.
~ Edited from Wikipedia and AnimeGames