Yes he does
In the film Treebeard talks about how Saruman used to walk through through Fangorn. Implying a familiarity at least from Treebeards point of view, as Treebeard knows enough about Saruman (including his name) to suggest this.
From The Two Towers Film
Treebeard: "There was a time, when Saruman would walk in my woods, but
now, he has a mind of metal and wheels."
And fom the book The Two Towers, Treebeard confirms a relationship more definitively with:
'But Saruman now! Saruman is a neighbour: I cannot overlook him. I must do something, I suppose. I have often wondered lately what I should do about Saruman.'
'Who is Saruman?' asked Pippin. 'Do you know anything about his history?'
'Saruman is a Wizard,' answered Treebeard.... '[He] was reckoned great among them, I believe. He gave up wandering about and minding the affairs of Men and Elves,... a very long time ago: and he settled down at... Isengard.... He was very quiet to begin with, but his fame began to grow. He was chosen to be head of the White Council, they say; but that did not turn out too well. I wonder now if even then Saruman was not turning to evil ways. But at any rate he used to give no trouble to his neighbours. I used to talk to him. There was a time when he was always walking about my woods. He was polite in those days, always asking my leave... and always eager to listen. I told him many things that he would never have found out by himself; but he never repaid me in like kind....'
Here it states Treebeard and Saruman talked. If you talk with someone you must know them.
As to why he didn't properly defend against them.
- He was in Orthanc, a tower with its strong walls
"A great ring-wall of stone, like towering cliffs, stood out from the shelter of the mountain-side, from which it ran and then returned again. One entrance only was there made in it, a great arch delved in the southern wall. Here through the black rock a long tunnel had been hewn, closed at either end with mighty doors of iron. They were so wrought and poised upon their huge hinges, posts of steel driven into the living stone, that when unbarred they could be moved with a light thrust of the arms, noiselessly. One who passed in and came at length out of the echoing tunnel, beheld a plain, a great circle, somewhat hollowed like a vast shallow bowl: a mile it measured from rim to rim." - The Two Towers - The Road to Isengard
He would have felt safe (and was infact safe as the Ents had to besiege the tower).
- He underestimated the anger of the Ents. He (his Orcs) had been cutting down vast swathes of Fangorn forest down with no reprisal, and, had Merry and Pippin not met Treebeard he may have been safe in this assumption.
- He thought that the Ents had all become more 'Treeish' to the point that they would not be able to do anything anyway.