As far as I know, the rules for Trial by Combat in the Game of Thrones universe are never clearly explained. Historically, there have been many different forms of trial by combat, with various rules governing their use and outcome.
Most likely, Martin is basing his combat trials on what the Norman's called "wager of battle", which was common in England in the Middle Ages. This type of combat was used in both criminal cases (like Tyrions) as well as in civil cases, such as land disputes. In the criminal case, the battle was not a contest to determine which side was right; rather, it was a contest to determine if the defendant was guilty. The crown's champion could not "lose" a trial by combat in the normal sense, he could only "fail to win".
According to the rules in place for most of the Middle Ages, a defendant won a trial by combat by surviving the duration of the combat, which would usually last until one combatant could no longer continue, or one combatant yielded, or from dawn to dusk. If the defendant was still alive and able to fight at the end of the combat, they were judged innocent. If they were defeated, killed, or yielded, they were judged guilty.
Of particular note is the fact that the condition of the crown's champion does not actually factor in to the final outcome, it only factors into the decision to end the trial prematurely. As long as the crown's champion can continue to fight, the trial continues, and the defendant can lose.
In Tyrion's case:
The Mountain was clearly not incapacitated by Oberyn, and did not yield. The fact that he dies anyway is irrelevant.