As others have said, each major wizard identifies themselves as such by taking the image from previous ones - especially the long white beard and flowing robes.
However the origins of the wizard image are from "wise men", the people in a tribe or group who dedicated their lives to gaining wisdom and information, and who used their learned skills to help the tribe.
The long grey beard is a visual sign of this, and a natural effect of spending longer reading books than shaving - it is an indication of age, especially in times past when reaching 40 was an achievement. It also gives a greater sense of "age" and "wisdom" to the wizard.