I guess that any kind of dark magic would have got him in trouble, though at different levels (from school detention to wizard prison).
Being so talented at potions, as Slughorn recognizes ("why, not even you, Severus!"), Snape might have experimenting with potions to deal with friends and foes. For example:
- Healing potions (for himself, after most encounters with Sirius/James).
- Potions to improve concentration during exams (for himself/friends).
- Love potions (for Lily, assuming he had a physical, adolescent interest first, before falling in love with her).
- Poisons (for foes).
Not all of these are necessarily dark, but some of them are certainly illegal. Even a simple potion for concentration might be considered cheating, just like a self-correcting quill. And a love potion is quite dangerous and can certainly be used for dubious objectives bordering on illegality (see: Merope Gaunt and/with Tom Riddle Sr.).
Now, poisons might have been some kind of a favorite hobby for him. He had the talent, the enemies and the intention, so here's where I guess he put more effort. In my opinion, he studied and developed dark magic for this purpose.
However, it is not clear whether he actually used it on somebody, just as it's unclear that he ever used Sectum Sempra, because there's not much information on how spells are created (I guess at least you need to practice on somebody else). James and Sirius were his karma, but he did enjoy the company of bullies and "evil" people (as Lily tells him), so he most certainly might have had enemies other than the Potter gang, and he must have tried these dark potions on them.
This is only a wild guess, but as he did invent Sectum Sempra as a teenager (and marked it as being "for enemies"), I think it's likely he tried some other stuff as well. He would have been only too happy to poison James Potter just before a Quidditch match so he would fall off his broom five minutes into the game.