Simply put - if a Dothraki army attacks, they are defeated, and they retreat, then they cut their braids.
Speaking of the book series, I can think of one reference for Dothraki cutting braids:
A Storm of Swords, chapter 8 - Daenerys.
Ser Jorah discusses Unsullied warriors with Daenerys and he recalls the events of the Battle of Qohor.
The Dothraki charged. The Unsullied locked their shields, lowered their spears, and stood firm. Against twenty thousand screamers with bells in their hair, they stood firm.
Eighteen times the Dothraki charged, and broke themselves on those shields and spears like waves on a rocky shore. Thrice Temmo sent his archers wheeling past and arrows fell like rain upon the Three Thousand, but the Unsullied merely lifted their shields above their heads until the squall had passed. In the end only six hundred of them remained . . . but more than twelve thousand Dothraki lay dead upon that field, including Khal Temmo, his bloodriders, his kos, and all his sons. On the morning of the fourth day, the new khal led the survivors past the city gates in a stately procession. One by one, each man cut off his braid and threw it down before the feet of the Three Thousand.
As for TV series - HBO's official viewer's guide website gives following description of the Dothraki.
The Dothraki are a nomadic band of horse lords who live in Essos on the plains beyond the Free Cities. They travel in packs known as khalasars, led by khals. A khal's strength can be determined by the length of his hair, as the khals only cut their hair after a defeat.