It is most likely that none of the public at that time knew.
It is never stated whether anyone else, save Grindelwald + Voldemort, knew that Dumbledore had the wand. It is only stated that people knew about the wand, not of the possessor.
Your point on the uniqueness of the Elder wand is correct, but, it is mentioned in the quote that these unique characteristics of this wand is recognised by those 'learned in wandlore'. (Emphasis mine)
"Oh yes, yes, it is perfectly possible to trace the wand's course through history. There are gaps, of course, and long ones, where it vanishes from view, temporarily lost or hidden; but it always resurfaces. It has certain identifying characteristics that those who are learned in wandlore recognise [...]
Therefore, only those learned in wandlore seem to be able to recognise the Elder wand. I believe you're taking the film references of the wand to be canonical. The films show very distinct characteristics of the wand: e.g. the grooves and the unique shape. This is not so in the canonical universe, as mentioned above in the quote.
As you have stated, both Grindelwald and Voldemort knew that Dumbledore had the wand. Grindelwald, since Dumbledore had defeated him. Voldemort only learnt that after he visited Grindelwald in his prison. Other than these 2, no other individual was stated to have recognised Dumbledore's wand.
Ollivander most certainly did not:
'It was a rumour,' whispered Ollivander. 'A rumour, years and years ago, long before you were born! I believe Gregovoritch himself started it. You can see how good it would be for business: that he was studying and duplicating, the qualities of the Elder Wand!'
The fact that Ollivander calls it a 'rumour' and the use of the word 'believe' proves that he wasn't sure if Greg actually had the wand; so he definitely couldn't be sure that Dumbledore had it.
Xenophilius, as well, is aware of Wand's existence, but unaware that Dumbledore had it:
'So where do you think the Elder Wand is now?' asked Ron.
'Alas, who knows?' said Xenophilius, [...]
Hence, these 2 people (including Greg), were aware of the wand's existence, but never knew that Dumbledore possessed it.
Xenophilius also mentioned:
'[...] The trail goes cold with Arcus and Livius. Who can say which of them really defeated Loxias, and which took the wand? And who can say who may have defeated them? History, alas, does not tell us.'
For those who had not heard Greg's rumour of him possessing the wand(Xeno), they would have no idea that the Elder wand was with Grindelwald, and subsequently Dumbledore.
It is likely that does who were present at the Grindelwald VS Dumbledore battle saw Dumbledore taking the wand, but it is never stated.
Also, don't forget that the Elder wand was found after Dumbledore's death and laid with him inside his tomb. The fact that it was the Elder wand was never brought up, as presumably known recognised that wand. Explained here: Whose idea was it to put the Elder Wand in Dumbledore's Tomb in the first place, instead of somewhere considerably less likely or vulnerable?
Lastly, I would also like to add that the Elder wand wasn't exactly 'famous' in Harry's time. The Elder wand was known by multiple names, and hence people couldn't confirm that these wands were all the same. (This is better explained by I Love You's answer.)