Well, the Avada Kedavra, Imperius, and Cruciatus Curses were made illegal by the Ministry of Magic in 1717 and classified as Unforgivables at that time via legislation, "with the strictest penalties attached to their use." (Tales of Beedle the Bard - Page 142 - Collector Edition). So, technically, it's due to legislation.
However, legislation is always based on the will of a group of people (I won't say 'all'), so at some point before 1717, concern regarding Avada Kedavra, Imperius, and Cruciatus reached a boiling point. Perhaps there was a particular incident or series of incidents that served as a catalyst for the legislation and categorisation of the Unforgivables, but it's clear that these three spells in particular were problematic.
Evil is subjective. Why the Unforgivables are classified as such while Obliviate or Sectumsempra or an entrail-exploding curse are not included in that category is unclear from canon. Personally, I think that wiping memories or transfiguring someone against their will into an animal is morally reprehensible in general. There are always exceptions to the rule, where the use of a particular spell or curse might be deemed justified. It depends what POV you're looking at a situation from. Did Potter fans get all up in arms when Harry used the Imperius Curse in Deathly Hallows? No. Because they believed in him and his mission. But no-one likely felt sympathy for Barty Crouch for keeping Barty Crouch Jr. under the Imperius Curse for the however many years that he did, because his motivations were not ultimately selfless. That's what makes evil so insidious, I think. Like humour, it's so often subjective.
(NOTE: No cat was actually mowed over during the production of this post)