In his 1954 letter on The Istari (Wizards), Tolkien comments on the Heren Istarion (Order of Wizards) being larger than 5 but only 5 came to the North West of Middle-earth. It is worth noting that this is the only time an unknown number for the Order is mentioned. It is unclear how long this idea lasted or when it had formed, only that it existed in 1954.
Of this Order the number is unknown; but of those that came to the North of Middle-earth, where there
was most hope (because of the remnant of the Dúnedain and of the Eldar that abode there), the chiefs were five.
The Unfinished Tales - Part IV, The Istari
In the context of the question, however, we get no explicit information that there were any more true wizards than the number we know of. This would suggest that for the above quotes, Tolkien had initially thought of the knowledge of wizards being far more common than we are later led to believe, with most people having been exposed to the 3 that roamed the North-West of Middle-earth and had been in contact with them often. This may have given the impression of "large numbers" as due to their long age, stories would pass down of encounters with wizards, Hobbits likely believing they were seeing different wizards.
The Hobbit and early writings of The Fellowship of the Ring:
There are instances in the Hobbit, which suggest that wizards may have been more common. While unclear whether this is with respect to there being wizards in an earlier form of the Legendarium, or because the Hobbit may have originally been part of a separate story, that was only brought into the Legendarium after it's success and the request for a follow-up. This combined with the path of the early drafts of the Fellowship being a close sequel to the Hobbit seems to have left remnants of an older idea of Tolkien's.
Even the good plans of wise wizards like Gandalf and of good friends like Elrond go astray sometimes when you are off on dangerous adventures over the Edge of the Wild...
The Hobbit - Chapter IV: Over Hill and Under Hill
Yes; not a bad fellow as wizards go, I believe.
ibid. - Chapter VII: Queer Lodgings
These two quotes seem to suggest that wizards may not have existed in large numbers but there were certainly more than just the two, Radagast and Gandalf, that we encounter. During the writing of the Hobbit, Tolkien seemed to entertain the idea that certain members of the Elder Race (Elves) may have been considered wizards or having wizardly traits (History of the Hobbit, John D. Rateliff)
It appeared that Gandalf had been to a great council of the white wizards, masters of lore and good magic...
ibid. - Chapter XIX: The Last Stage
It is true that for ever after he remained an elf-friend, and had the honour of dwarves, wizards, and all such folk as ever passed that way...
Both of the above were spoken by Bilbo in the final chapters of The Hobbit, the former mentions a "great council of white wizards"; what would later become known as The White Council. We know from the LotR that the Council was made up (at least) of two Wizards and three Elves (with the possible inclusion of Radagast in the "great council"), whether this suggests that the original idea contained at least these 3 Elves, or whether there were other Wizards or if the Elves were the ‘Wizards’ isn't outlined. The final quote, however, does suggest that the Elves remained separate from the Wizards and we can therefore speculate that when the original host of Wizards became 5, Tolkien replaced the Wizards in the council with Elves.
And certainly it was from Bree that the art of smoking the genuine weed spread in the recent centuries among Dwarves and such other folk, Rangers, Wizards, or wanderers...
The Fellowship of the Ring - Prologue
The above seems to be the only other relevant mention of wizards in the LotR which seems to be relevant to wizards other than the 5 that once ventured through the North-West of Middle-earth. It can however be written off as being purely about the 5 which made up the Istari as the three we know most of at least 2 were regular smokers of pipe-weed, and it could be supposed that Radagast had similar habits based on his expertise.
The People's of Middle-earth
ñolmo a wise person; ingólemo one with very great knowledge, a 'wizard'. This last word was however archaic and applied only to great sages of the Eldar in Valinor (such as Rúmil). The wizards of the Third Age -emissaries from the Valar - were called Istari 'those who know'.
The above is the only other mention I could find relevant to wizards possibly other than the 5 outside the core 4 books and the Unfinished Tales. The preceding discussion is based on the definition of the Noldorin prefix "Ñolo". From such, the words for "wise person" and "one with very great knowledge" are derived. From the latter we could make the guess that in earlier forms of the legendarium, those of very great knowledge such as Cirdan or Elrond would be thought of as wizards. Similar to the "great sages" of the Eldar in Valinor.