I've thought about this a lot, and I believe there's some sort of magic woven into the cloak by Ignotus Peverell that ensures the cloak will stay within the family line.
The first clue is in the Tales of Beedle the Bard:
"But though Death searched for the third brother for many years, he was never able to find him. It was only when he had attained a great age that the youngest brother finally took off the Cloak of Invisibility and gave it to his son. And then he greeted Death as an old friend, and went with him gladly, and, equals, they departed this life."
Although this story is from a children's fable, some believe it has factual roots -- quirky Xenophilius, of course, as well as Dumbledore and Grindelwald among them. The fable chronicles not only the items of legend, but also the manner in which they change possession.
Nega-Dumbledore later states:
“The Cloak, as you know now, traveled down through the ages, father to son, mother to daughter, right down to Ignotus’s last living descendant, who was born, as Ignotus was, in the village of Godric’s Hollow.”
None of this is even remotely conclusive proof, of course. But it does lend evidence to explain how, despite Harry's cavalier, negligent treatment of the cloak, he never lost ownership of it. Each time Harry and his cloak were parted, they were infallibly reunited. Imagine if the cloak were a hundred dollar bill. Would the following statements remain true?
In book 1, Harry leaves the cloak at the top of the tower after handing off Norbert to Charlie's associates. Dumbledore returns it later.
In the Prisoner of Azkaban, he tosses it aside at the base of the Whomping Willow. Snape helpfully brings it to Harry in the Shrieking Shack. Harry, determined to lose the thing, leaves it behind yet again. Lupin eventually retrieves it and returns it to Harry.
When Harry is captured by Umbridge in her school office, the cloak is left in a heap on the floor yet again. Nothing more is said of the cloak for the rest of the book; but apparently, miraculously, all the Slytherins ignore it and it is somehow returned to Harry's possession by the next book. Despite his most fervent efforts, Harry can't seem to rid himself of the cloak.
On the train in Half-Blood Prince, Harry tumbles out of the luggage rack and loses his cover. Malfoy graciously replaces the cloak over Harry, tucking him in like a caring mother. Later, the cloak is again tossed aside at the top of the Astronomy tower, where it remains all but forgotten until the next book.
Additionally, the cloak is not summonable. In Deathly Hallows:
“Accio Cloak!” roared one of the Death Eaters.
Harry seized its folds, but it made no attempt to escape: The Summoning Charm had not worked on it.
I believe the reason for this is two-fold. Obviously, an anti-summoning charm would add to the perfection of the invisibility offered by the cloak. Secondly, it would be one of possibly many enchantments preventing the cloak from being stolen.
There are situations where others borrow the cloak -- Snape, at the Whomping Willow; Ron and Hermione, scouting the Ministry of Magic; and Dumbledore from James as he was investigating the Deathly Hallows. But it's clear that they all considered Harry to be the owner of the cloak. If Draco had taken the cloak against Harry's wishes, I'm guessing he would do so not intending to give it back -- which is what the hypothesized enchantments would prevent. Evidence suggests that anyone who comes across the cloak has a compulsion to intend to return it to its owner.
