I actually answered this on a duplicate and though much of what I said overlaps with the rightfully accepted answer to this original posting of the question, DVK did leave out one thing:
Kids with a less than loving home-life, on the outskirts with peers . . . kids that don't feel they fit in or belong anywhere can be desparate to "fit in." to Fit in somewhere - anywhere. Voldemort himself was one of these kids at one time. Dumbledore did try to make Hogwart's feel like home and Voldy did develop some relationships with peers - unfortunately, he became the leader of what turned out to be a pretty horrific gang and I think was far more violently inclined and angry than Dumbledore ever anticipated.
In addition to the wanting revenge factor, Snape fits into this category of kids too - doesn't really belong anywhere, so when the DE's welcome him, it is just too much to refuse - No one else had ever made him feel accepted (except perhaps, Lily - until she chose James). One could argue a little of this was going on for Pettigrew too. Although he was part of the Marauders, he was a bit of a hanger-on and even if the other three were very loving and inclusive, if Pettigrew didn't believe he was up to snuff, nothing they did would have mattered - he didn't feel he was a full member.
Some of Voldy's followers were friends with him in school when he was still more normal. Initially, they probably just had some things in common - such as being in Slytherin, wanting only pure-bloods to be at Hogwart's, etc. etc. It probalby started out a bit like Crabbe and Goyle's relationship with Draco Malfoy. They were drawn to Malfoy because of his confidence and swagger. I'd imagine Tom Riddle would have had real confidence and swagger even from day one. Some of his most beloved and original followers were probably simply drawn to him at the beginning of things. For naive boys/girl (Bellatrix?) the ultimate consequences of "bullying" others didn't really occur to them. As Riddle turned into Voldemort, they were already part of the circle and stayed (because of a crazy obsession or) despite reservations and maybe wanting to leave. Malfoy Sr. strikes me as being the type of character that would fit this role. He had some common beliefs, was drawn to the charisma Riddle/Voldemort exuded, joined up and then probably regretted it later (at least to some degree) but stayed in for fear of what would happen if he didn't (and because he still believed in the underlying ideas that made him believe in Voldemort in the first place.