Pettigrew was nearly a Slytherin
Pettigrew certainly had many traits characteristic of Slytherin. According to Pottermore, he was a Hatstall between Gryffindor and Slytherin, meaning that he was, in the Hat's estimation, about evenly balanced between them:
The only true Hatstalls known personally to Harry Potter were Minerva
McGonagall and Peter Pettigrew. The former caused the hat to agonise
for five and a half minutes as to whether Minerva ought to go to
Ravenclaw or Gryffindor; the latter was placed in Gryffindor after a
long deliberation between that house and Slytherin. The Sorting Hat,
which is infamously stubborn, still refuses to accept that its
decision in the case of the latter may have been erroneous, citing the
manner in which Pettigrew died as (dubious) evidence.
The phrasing here even suggests that the Hat may have made a mistake, referring to Pettigrew's actions toward Harry as "(dubious) evidence" of his suitability for Gryffindor.
The Sorting Hat does not use any unusual magic to determine students' aptitudes. All it has is Legilimency.
Talks, sings, uses Legilimency
It can only determine someone's disposition from what is in their head at that time: their thoughts, their memories. Despite the Sorting Hat's stubborn insistence otherwise, it is quite possible for people to change between their Sorting and their death, a fact that Dumbledore recognized:
“Karkaroff’s Mark is becoming darker too. He is panicking, he fears
retribution; you know how much help he gave the Ministry after the
Dark Lord fell.” Snape looked sideways at Dumbledore’s crooked-nosed
profile. “Karkaroff intends to flee if the Mark burns.”
“Does he?” said Dumbledore softly, as Fleur Delacour and Roger Davies
came giggling in from the grounds. “And are you tempted to join him?”
“No,” said Snape, his black eyes on Fleur’s and Roger’s retreating
figures. “I am not such a coward.”
“No,” agreed Dumbledore. “You are a braver man by far than Igor
Karkaroff. You know, I sometimes think we Sort too soon. . . ”
—Harry Potter and the Deathly Hallows
When Pettigrew was Sorted, he was balanced on the edge between Gryffindor and Slytherin. Perhaps he had enough bravery, or potential for bravery, to develop into a true Gryffindor, much as Neville Longbottom did. But what happened after his Sorting? He was taken in by James's group (and James was a bit of a bully). He was pushed to its periphery and constantly mocked by the others. And then a war with Voldemort started, which could have pushed his fear to new levels. In the end, the Sorting Hat simply could not have foreseen all this.
So Pettigrew did not really display much bravery in the main series. But no surprise: this was decades after the Sorting Hat, just barely, though that he had an aptitude for Gryffindor. In the ensuing period, he had developed in quite a different direction.