This is largely covered in "The Unfinished Tales"
Basically there were (initially) only 5 of the Istari.
Wizard is a translation of Quenya istar (Sindarin ithron): one of the members of an ‘order’ (as they called it), claiming to possess, and exhibiting, eminent knowledg1e of the history and nature of the World. The translation (though suitable in its relation to ‘wise’ and other ancient words of knowing, similar to that of istar in Quenya) is not perhaps happy, since the Heren Istarion or ‘Order of Wizards’ was quite distinct from the ‘wizards’ and ‘magicians’ of later legend; they belonged solely to the Third Age and then departed, and none save maybe Elrond, Círdan, and Galadriel discovered of what kind they were or whence they came.
They spent a lot of time hanging around with important and powerful people
Among Men they were supposed (at first) by those that had dealings with them to be Men who had acquired lore and arts by long and secret study. They first appeared in Middle-earth about the year 1000 of the Third Age, but for long they went about in simple guise, as it were of Men already old in years but hale in body, travellers and wanderers, gaining knowledge of Middle-earth and all that dwelt therein, but revealing to none their powers and purposes. In that time Men saw them seldom and heeded them little.
As Sauron's power rose, people paid a hell of a lot more attention to them
But as the shadow of Sauron began to grow and take shape again, they became more active, and sought ever to contest the growth of the Shadow, and to move Elves and Men to beware of their peril. Then far and wide rumour of their comings and goings, and their meddling in many matters, was noised among Men;
At which point it was noticed that they weren't aging. This gave them celebrity status since they've now had hundreds of years to become well known
and Men perceived that they did not die, but remained the same (unless it were that they aged somewhat in looks), while the fathers and sons of Men passed away. Men, therefore, grew to fear them, even when they loved them, and they were held to be of the Elven-race (with whom, indeed, they often consorted).