Generally wizards did not like doing things the muggle way. Voldemort was a wizard supremacist and hated muggles. He even changed his name because he was named after a muggle.
“It was a name I was already using at Hogwarts, to my most intimate friends only, of course. You think I was going to use my filthy Muggle father’s name forever? I, in whose veins runs the blood of Salazar Slytherin himself, through my mother’s side? I, keep the name of a foul, common Muggle, who abandoned me even before I was born, just because he found out his wife was a witch? No, Harry — I fashioned myself a new name, a name I knew wizards everywhere would one day fear to speak, when I had become the greatest sorcerer in the world!”
Harry Potter and the Chamber of Secrets, Chapter 17, The Heir of Slytherin
Voldemort always wanted to be different and superior to others. He cant be superior if he uses knife or gun.
“I knew I was different,” he whispered to his own quivering fingers. “I knew I was special. Always, I knew there was something.”
“Well, you were quite right,” said Dumbledore, who was no longer smiling, but watching Riddle intently. “You are a wizard.”
Harry Potter and the Half-Blood Prince, Chapter 13, The Secret Riddle
He might have wanted to use Harry for making his sixth Horcrux.
“Well, it is inadvisable to do so,” said Dumbledore, “because to confide a part of your soul to something that can think and move for itself is obviously a very risky business. However, if my calculations are correct, Voldemort was still at least one Horcrux short of his goal of six when he entered your parents’ house with the intention of killing you. He seems to have reserved the process of making Horcruxes for particularly significant deaths. You would certainly have been
that. He believed that in killing you, he was destroying the danger the prophecy had outlined. He believed he was making himself invincible. I am sure that he was intending to make his final Horcrux with your death. As we know, he failed. After an interval of some years, however, he used Nagini to kill an old Muggle man, and it might then have occurred to him to turn her into his last Horcrux. She underlines the Slytherin connection, which enhances Lord Voldemorts mystique; I think he is perhaps as fond of her as he can be of anything; he certainly likes to keep her close, and he seems to have an unusual amount of control over her, even for a Parselmouth.”
Harry Potter and the Half-Blood Prince, Chapter 23, Horcruxes
He would have found it distasteful to use common muggle way of killing. Simple answer to your question is his ego would not have allowed him to use muggle ways.
But I knew, too, where Voldemort was weak. And so I made my decision. You would be protected by an ancient magic of which he knows, which he despises, and which he has always, therefore, underestimated - to his cost. I am speaking, of course, of the fact that your mother died to save you.
Harry Potter and the Order of the Phoenix, Chapter 36, The Only One He Ever Feared
And most importantly he would not have expected the spell to backfire on him. Voldemort had killed lots of people and did not face similar problem. So it was natural for him to use a curse.