Saruman did meet Frodo once before, on the route from the Gap of Rohan to Rivendell on their return journey:
On the sixth day since their parting from the King they journeyed through a wood climbing down from the hills at the feet of the Misty Mountains that now marched on their right hand. As they came out again into the open country at sundown they overtook an old man leaning on a staff, and he was clothed in rags of grey or dirty white, and at his heels went another beggar, slouching and whining.
Furthermore, they did speak to each other:
'So you have come to gloat too, have you, my urchins?' he said. 'You don't care what a beggar lacks, do you? For you have all you want, food and fine clothes, and the best weed for your pipes. Oh yes, I know! I know where it comes from. You would not give a pipeful to a beggar, would you?'
'I would, if I had any,' said Frodo.
Despite this, there doesn't actually seem to be much in Saruman's speech that's specifically aimed at Frodo, and it's worth quoting it in full for examination:
'You have grown, Halfling,' he said. 'Yes, you have grown very much. You are wise, and cruel. You have robbed my revenge of sweetness, and now I must go hence in bitterness, in debt to your mercy. I hate it and you! Well, I go and I will trouble you no more. But do not expect me to wish you health and long life. You will have neither. But that is not my doing. I merely foretell.'
Notable here is that Saruman addresses Frodo as "Halfling", and aside from his foretelling at the end (which, it seems to me, need not require specific knowledge of Frodo to make), the content of his speech could just as easily have been directed at any of the others.
It therefore appears to have been merely a reply to Frodo, and the cause of the reply was just that Frodo was the one who had previously spoken. If it had been any of the others who had previously spoken, there seems no reason why Saruman would not have made a similar, if not identical, reply.
Again, despite all of this, Saruman does appear to have a talent for divining people's names, origins and motivations from merely speaking with them. We see this in the chapter The Voice of Saruman, where he opens with:
Two at least of you I know by name. Gandalf I know too well to have much hope that he seeks help or counsel here. But you, Theoden Lord of the Mark of Rohan are declared by your noble devices, and still more by the fair countenance of the House of Eorl.
Following Gimli's subsequent interjection, he then replies:
'Peace!' said Saruman, and for a fleeting moment his voice was less suave, and a light flickered in his eyes and was gone. 'I do not speak to you yet, Gimli Gloin's son,' he said. 'Far away is your home and small concern of yours are the troubles of this land. But it was not by design of your own that you became embroiled in them, and so I will not blame such part as you have played – a valiant one, I doubt not.'
It's notable here that despite not knowing, and not having been told, Gimli's name, Saruman still addresses him by name and knows of his homeland.
In addition to all of this, it should be remembered that Saruman had many spies throughout Middle-earth, at least one of whom had a previous encounter with Frodo: the "squint-eyed southerner" in Bree, who is confirmed by Unfinished Tales to have been a spy of Saruman's:
Some while ago one of Saruman's most trusted servants (yet a ruffianly fellow, an outlaw driven from Dunland, where many said that he had Orc-blood) had returned from the borders of the Shire, where he had been negotiating for the purpose of "leaf" and other supplies ... He was the squint-eyed southerner at the Inn.
(Unfinished Tales, The Hunt for the Ring)
So in summary, Saruman's final words to Frodo may be seen as coming from a combination of four factors:
- Knowledge gleaned from his earlier meeting with Frodo,
- Generic words not specifically aimed at Frodo, but rather just aimed at the last Hobbit who spoke to him,
- The same divining talent he displayed in his earlier dealing with Gimli, and,
- Information obtained by means of his use of spies throughout Middle-earth.