Harry does not think he is a good swimmer.
Harry dreads swimming in the second Triwizard task:
He wasn’t a very good swimmer; he’d never had much practice. Dudley had had lessons in their youth, but Aunt Petunia and Uncle Vernon, no doubt hoping that Harry would drown one day, hadn’t bothered to give him any.
Goblet of Fire, Chapter 25: "The Egg and the Eye"
Before Hogwarts, Harry had no opportunity to learn sports.
The Dursleys have a long history of excluding Harry from fun extracurricular activities:
Every year on Dudley’s birthday his parents took him and a friend out for the day, to adventure parks, hamburger bars or the cinema. Every year, Harry was left behind with Mrs Figg, a mad old lady who lived two streets away. Harry hated it there.
Philosopher's Stone, Chapter 2: "The Vanishing Glass"
Dudley "encouraged" Harry to run.
Although Harry never had opportunity to learn sports skills, running from Dudley seems to have kept Harry in good aerobic condition:
On the other hand, he’d got into terrible trouble for being found on the roof of the school kitchens. Dudley’s gang had been chasing him as usual when, as much to Harry’s surprise as anyone else’s, there he was sitting on the chimney.
Philosopher's Stone, Chapter 2: The Vanishing Glass