In "The Prisoner of Azkaban", chapter 17, "Cat, Rat, and Dog":
Harry raised the wand. Now was the moment to do it. Now was the moment to avenge his mother and father. He was going to kill Black. He had to kill Black. This was his chance...
How was Harry going to kill him, had he tried to go through with it? It's only his third year, and I don't think he knows any killing curses yet.
Canonically speaking, I'm going to say that Harry wasn't going to kill Sirius at all. Yes, Harry was blind with anger and hurt, and he dreamed of avenging his parents by killing Sirius, but Harry had not been taught any deadly curses prior to Goblet of Fire and he didn't understand the emotional component of killing another human being. As Bellatrix Lestrange said to Harry, after he failed to cast the Cruciatus curse against Bella in Order of the Phoenix, "You've got to really mean it, Potter!" [Executing the Unforgivables, that is/paraphrase]
However, Harry's experience with any kind of magical killing curse prior to Goblet of Fire is limited to the following:
‘Then kill him, fool, and be done!’ screeched Voldemort.
Quirrell raised his hand to perform a deadly curse, but Harry by instinct, reached up and grabbed Quirrell’s face –
‘AAAARGH!’
Philosopher's Stone - pages 213-214 - Bloomsbury - chapter seventeen, The Man With Two Faces
The "deadly curse" isn't even named! Prior to Goblet of Fire, Harry had not heard of Avada Kedavra.
How Harry knew Quirrell was going to perform a deadly curse -- since Quirrell only raised his wand without saying anything -- is not explained. In canon, aside from Avada Kedavra, no other curse or spell is described as being deliberately lethal, except, I'd argue, Fiendfyre (And even then it's a stretch; it is lethal magic, but its foremost purpose is to burn and destroy; that it is lethal is incidental.).
Before all the talk about Harry wanting to kill Sirius in Prisoner of Azkaban, Harry is determined to hurt Sirius, to try and offload the hurt Harry himself felt due to his parents' death. Harry wanted Sirius to hurt, emotionally, as badly as he did.
He had forgotten about magic – he had forgotten that he was short and skinny and thirteen, whereas Black was a tall, fullgrown man. All Harry knew was that he wanted to hurt Black as badly as he could and that he didn’t care how much he got hurt in return ...
Prisoner of Azkaban - page 249 - Bloomsbury - chapter seventeen, Cat, Rat, and Dog
Harry was not trained in magic well enough to kill Sirius. If he were to actually try (Which I feel is against his characterization. Anger? Yes. Murder? No. Killing is something deep down that Harry would never want to do, although I don't think he regretted killing Voldemort.) I think he would have tried to physically kill Sirius. After all, the thought of magic had fled Harry's mind and he and Sirius had an initial physical altercation. Yes, Harry then pointed his wand at Sirius, but if magic had fled his mind, raising his wand could have merely been reflexive.
Anyhow, there are basically only two ways Harry could have killed Sirius and canon only allows for generalizations: Harry would have killed Sirius either by magic or through physical violence. Order of the Phoenix demonstrates Harry isn't capable of an Unforgivable at age 15, much less 13. There are no canon instances showing Harry physically beating anyone to the point of death. So it's a toss up.
Personally, I don't think Harry would have ever killed Sirius, especially at age thirteen. I do think canon ultimately supports this.
Just because magic is valid in the universe, it doesn't mean non-magical things couldn't be done.
As @DVK has commented, Harry could use Expelliramus spell to disarm Sirius (as a defensive act). Then, all he needed was to use a sharp stone or something similar to kill him. He was also able to lift a big piece of rock with magic to drop it on Sirius. Lifting Sirius to the killer tree outside was also an option.
The scene is somewhat ambiguous on that matter. On the one hand, there's the "forgotten about magic"-part, on thee other hand he seems quite determined at pointing his wand at Sirius' heart and going through with killing him (even despite the cat trying to protect him). I think the former can more likely be seen as referring to his initial attack however, while once he points his wand at him intent on finishing him off he must have had the intention of using magic.
There are several valis possibilities imo.
First, perhaps most likely, I think he was acting on impulse and emotion and hence, had he done so, might have used instinctive magic without any particular spell, similar to how he blew up his aunt.
Secondly, while Avada Kedavra is probably the only spell explicitely and exclusively made for that purpose, that doesn't make it the only spell capable of causing death when used in the right way. That he aims at the heart further supports this, I can think of a number of everyday or at least non-unforgivable spells that could cause a wound to a human, and ergo be fatal when applied to the right point. For example Diffindo, which he knows by the fourth book and thus could have potentially known at the end of book 3, and which if I remember correctly can cause wounds to humans.
Finally, although I don't think it likely, a sufficiently sturdy piece of wood is a perfectly valid stabbing weapon in itself, even without magical properties.
Perhaps, since he KNOWS spells, even though he doesn't learn the killing curse until year 4, (what do you think they have been learning in Professor Lupin's class when they weren't learning about magical creatures, huh?) he could use different spells. You know when he hit Severus Snape, it says that his scalp started bleeding, perhaps you could hit him with it enough times to kill him.
The scene (in the both the books and the movies) shows that Harry is acting on pure Adrenaline and/or rage, and that he really doesn't think of how to kill Sirius. From the movie, he was probably going for "choking"
Source: It appears as though Harry is about to try to strangle Sirius in the movie
"He [Harry] lunged forward-He had forgotten about magic...he wanted to hurt Black badly"
##Source: POA (pg 341) ## "Harry walked slowly nearer, his wand pointed straight at Black's heart...Harry stopped right above him, his wand pointing at Black's chest"
