Voldemort could have easily killed him without a fight in Goblet of Fire but seemed too proud to do so. Then in Deathly Hallows Voldemort just lets Harry surrender himself and kills him.
It's all about Voldemorts' ego: The only person that ever defeated him was Harry Potter and that is something that makes Voldemort think he looks weak in the eyes of his followers.
In Goblet of Fire he wants to prove, that Harry just survived out of pure luck and because of the protection of others and is in fact not stronger than Lord Voldemort.
“You see, I think, how foolish it was to suppose that this boy could ever have been stronger than me,” said Voldemort. “But I want there to be no mistake in anybody’s mind. Harry Potter escaped me by a lucky chance. And I am now going to prove my power by killing him, here and now, in front of you all, when there is no Dumbledore to help him, and no mother to die for him. I will give him his chance. He will be allowed to fight, and you will be left in no doubt which of us is the stronger
Harry Potter, Goblet of Fire, CHAPTER THIRTY-THREE
In Deathly Hallows Voldemort had already lost multiple times against Harry... He does not want to risk being beaten by Harry again and tries to directly kill him without any chance for Harry to defend (he thinks)
After the debacle at the end of Goblet of Fire, Voldemort changed his approach to dealing with Harry. Since Harry had now escaped Voldemort several times, Voldemort changed his focus to simply eliminating Harry in a quick and efficient manner, rather than trying to establish his dominance over him. This is evidenced when they meet again at the end of Order of the Phoenix:
Voldemort paid no attention.
“I have nothing more to say to you, Potter,” he said quietly. “You have irked me too often, for too long. AVADA KEDAVRA!”