Just to expand on @Christi's answer (which is correct: they were not being entirely rational because they are human, and because Angelus is very dangerous vampire):
Giles and Xander had very different motivations to not want to return Angel's soul. (This is typical of Xander, especially early on: he only gets what he wants when it happens to correspond to what someone more knowledgable or "important" also wants.)
Both of their issues focused on Buffy's relationship with him. If Angel's soul were returned, presumably Buffy and Angel would attempt to reconcile, and potentially resume their relationship. One of the key points Whedon has written into this entire series is that Buffy's still a human teenage girl, and has all the normal faults, failings, etc. of a human girl. In particular, this entire Angel episode is a pretty clear metaphor for the manipulative ex-boyfriend who only "plays nice" until he takes her virginity, then dumps her. It is very common for a person (especially a teenager) to form an irrationally deep bond with the first person they slept with, and want to maintain that relationship in the face of countless signs warning them not to. In this case, Buffy has the additional benefit of a good explanation for Angel's change in behavior, and reason to believe (correctly) that his behavior would change back if his soul could be restored.
For Giles, he doesn't want to see this happen for two reasons. Of course, the primary motivation is his hatred for Angelus, and what Angelus did to himself and Jenny. Also, he's playing his usual role as surrogate father figure for Buffy, and is pissed at Angel for the emotional trauma she went through. Giles certainly doesn't want to see Angel "rewarded" by having his soul restored and having his previous, not-happy-but-at-least-satisfying life back. But beyond that, he's a Watcher, and never really liked the idea of his Slayer befriending, much less dating, a vampire. (He similarly never fully accepts the idea of Spike being a member of the team.) In his mind, Angel reverting to Angelus is really just the world righting itself -- he's acting how vampires should act. The response should then be for Buffy to act how Slayers should act, and stake him. If Angel's soul were restored, Giles would have to face the same internal conflict he's been facing all along, only now his knowledge of Angelus's cruelty is more than just academic, it's personal. Angel has "proven" to Giles that, at his core, he's still just a vampire, and vampires do not deserve pity, they deserve death.
For Xander, it's a lot simpler. He has shown a very blatant infatuation with Buffy. He frequently made snide remarks question if Angel could be trusted, long after the rest of the "gang" had accepted him. Now, what had previously looked like Xander just being petty and jealous of Angel's relationship to Buffy, has been vindicated. In his mind, he "saw this coming". The last thing Xander wants is things back to the status quo.