We know that someone actually died while playing Quidditch. So why didn't anyone, such as Voldemort or Draco, just poke a quill in Harry's eye to kill him?
Since you mention Quidditch, recall that Voldemort/Quirrell did try to kill Harry while he was playing a game in The Philosopher's Stone by jinxing his broom.
Other attempts at assassination were also done, although most of them likely planned to use magic to do so, rather than stabbing him with a quill. Perhaps the most relevant to the phrasing of your question is that Barty Crouch Jr lured Harry to Moody's office in The Goblet of Fire to kill him personally.
Apart from that, Voldemort obviously tried to kill him in Goblet of Fire to use his blood in his own resurrection. Death Eaters also crashed the wedding of Bill and Fleur, presumably with the primary aim of killing Harry. Death Eaters also tried to kidnap Harry from the Hogwarts Express in the Half-Blood Prince.