Now, all of these don't qualify as formal duels or battle, but they all occur when at least one person is angry, scared or threatened.
**** Transfiguration used in battle ****
In Prisoner of Azkaban, Sirius Black, in his dog Animagus form, attacks and physically fights with Remus Lupin when Remus is in his werewolf form. Transforming into an Animagus is a form of Transfiguration.
In GoF, Mad-Eye Moody transfigures Draco Malfoy into 'Draco Malfoy, the amazing bouncing ferret' when Malfoy tries to curse Harry from behind.
In Deathly Hallows, Percy Weasley transfigured a Death Eater (Pius Thicknesse) into a sea urchin during the Battle of Hogwarts.
Hermione transfigures a tapestry into a stone wall with the spell Duro, and right after she does this a herd of galloping desks go by with McGonagall bringing up the rear - also a transfiguration.
**** Things that may be argued are transfiguration-like ****
In Chamber of Secrets, Draco Malfoy uses Serpensortia against Harry at the Dueling Club and produces a large snake. Hard to say if that's a transfiguration skill or not.
In Half-Blood Prince, Hermione sets a flock of conjured canaries after Ron by using the spell Oppugno.
**** Other methods used in battle which aren't jinxes/curses/protective spells/transfiguration ****
Question's subject asks specifically about Transfiguration, but the body merely talks about "dueling with magic other than curses, jinxes, and protection charms". So in case the body was more correct, this section is included.
In Philosopher's Stone, in the final scenes as Harry, Ron, and Hermione make their way through the various challenges protecting the Philosopher's Stone. They use both broom flying skills, magic chessplaying skills, and logic.
When Harry faces off with Professor Quirrell, Harry uses his own touch -- just his touch -- to defeat Quirrell, and the magic behind that is his mother's love.
In Chamber of Secrets, Harry uses his Parselmouth skill to get rid of the snake Malfoy conjired.
Also, Harry killed the Basilisk with the Sword of Gryffindor and had some assistance from Fawkes, a magical bird, who blinded the Basilisk.
In Order of the Phoenix, after Slytherin loses a Quidditch match to Gryffindor, Harry and George Wasley end up rushing Draco and attacking him physically. Later, McGonagall castigates Harry for "engaging in Muggle dueling".
In Deathly Hallows, in Battle of Hogwarts:
Neville Longbottom and Luna Lovegood bring Venomous Tentacula to use against the Death Eaters during the Battle of Hogwarts, and Snargaluff Pods as well.
Professor Trelawney throws crystal balls as weapons.
Neville uses the Sword of Gryffindor on Nagini.
Also, as far as edged weapons, Bellatrix throws a dagger at Harry (killing Dobby instead) earlier in the book.