Glasses as a Symbol
In most stories, glasses symbolize intelligence, foresight into the future, and enlightenment. But glasses also mean that if they are taken away, the person who needs them will be either hobbled or in need of new glasses . . . but none of these apply as a "vulnerability".
Relatively long answer, if unwilling to read, go to the conclusion.
Vulnerability Related to Glasses
Disclaimer: I do not know if JK Rowling meant this as I cannot find it in an interview
From my reading, I came to believe that Harry's glasses represented more than just glasses. In Diagon Alley when Harry's glasses break, he is alone (or as alone as he will be at that moment).
Harry waited for a minute in case he came back, then, quietly as he could, slipped out of the cabinet, past the glass cases, and out of the shop door.
Clutching his broken glasses to his face, Harry stared around.
He had emerged into a dingy alleyway that seemed to be made up entirely of shops devoted to the Dark Arts. The one he’d just left, Borgin and Burkes, looked like the largest, but the opposite was a nasty window display of shrunken heads and, two doors down, a large cage was alive with gigantic black spiders. Two shabby-looking wizards were watching him from the shadow of a doorway, muttering to each other. Feeling jumpy, Harry set off, trying to
hold his glasses on straight and hoping against hope he’d be able to find a way out of there.
An old wooden street sign hanging over a shop selling poisonous candles told him he was in Knockturn Alley. This didn’t help, as Harry had never heard of such a place. He supposed he hadn’t spoken clearly enough through his mouthful of ashes back in the Weasleys’ fire. Trying to stay calm, he wondered what to do.
Harry Potter and the Chamber of Secrets
The other times that his glasses break, Harry is usually alone (or feeling alone), such as when Dudley and his gang beat him up.
Perhaps it had something to do with living in a dark cupboard, but Harry had always been small and skinny for his age. He looked even smaller and skinnier than he really was because all he had to wear were old clothes of Dudley’s, and Dudley was about four times bigger than he was. Harry had a thin face, knobbly knees, black hair, and bright green eyes. He wore round glasses held together with a lot of Scotch tape because of all the times Dudley had punched him on the nose. The only thing Harry liked about his own appearance was a very thin scar on his forehead that was shaped like a bolt of lightning. He had had it as long as he could remember, and the first question he could ever remember asking his Aunt Petunia was how he had gotten it.
Harry Potter and the Philosopher's Stone
The vulnerability I believe JK Rowling to be saying is Harry's friendships/relations to other characters. Harry's glasses are usually broken or gone when he is alone. The glasses seem to be a symbol of Harry's friendships and how delicate they are. With Voldemort rising, everything Harry cares about is in danger and can be destroyed in a moment - just like his glasses.
Besides being destroyed, Harry will do anything to save his friends, as seen in book 7 where he chooses to sacrifice himself like his mother did. This is exploited by Voldemort (which backfired on him, so this is both a vulnerability and a strength of Harry's).