Yes he does
In the film Treebeard talks about how Saruman used to walk through through Fangorn. Implying a familiarity at least from Treebeards point of view, as Treebeard knows enough about Saruman (including his name) to suggest this.
From The Two Towers Film:
Treebeard: "There was a time, when Saruman would walk in my woods, but now, he has a mind of metal and wheels."
And from the book The Two Towers, Treebeard confirms a relationship more definitively with:
'But Saruman now! Saruman is a neighbour: I cannot overlook him. I must do something, I suppose. I have often wondered lately what I should do about Saruman.'
'Who is Saruman?' asked Pippin. 'Do you know anything about his history?'
'Saruman is a Wizard,' answered Treebeard.... '[He] was reckoned great among them, I believe. He gave up wandering about and minding the affairs of Men and Elves,... a very long time ago: and he settled down at... Isengard.... He was very quiet to begin with, but his fame began to grow. He was chosen to be head of the White Council, they say; but that did not turn out too well. I wonder now if even then Saruman was not turning to evil ways. But at any rate he used to give no trouble to his neighbours. I used to talk to him. There was a time when he was always walking about my woods. He was polite in those days, always asking my leave... and always eager to listen. I told him many things that he would never have found out by himself; but he never repaid me in like kind....'
— The Two Towers - Treebeard
Here it states Treebeard and Saruman talked, Saruman even asked Treebeard's permission in those early days, showing he had some measure of respect for the Ent. As Saruman's power / arrogance grew this respect seems to have waned until he no longer bothers with Treebeard at all.
As to why he didn't properly defend against them, some suppositions:
He was in Orthanc, a tower with its strong walls
"A great ring-wall of stone, like towering cliffs, stood out from the shelter of the mountain-side, from which it ran and then returned again. One entrance only was there made in it, a great arch delved in the southern wall. Here through the black rock a long tunnel had been hewn, closed at either end with mighty doors of iron. They were so wrought and poised upon their huge hinges, posts of steel driven into the living stone, that when unbarred they could be moved with a light thrust of the arms, noiselessly. One who passed in and came at length out of the echoing tunnel, beheld a plain, a great circle, somewhat hollowed like a vast shallow bowl: a mile it measured from rim to rim."
— The Two Towers - The Road to Isengard
and the tower itself appeared to be made of even stronger material
Many of the Ents were hurling themselves against the Orthanc-rock; but that defeated them. It is very smooth and hard.
Some wizardry is in it, perhaps, older and stronger than Saruman's. Anyway they could not get a grip on it, or make a crack in it; and they were bruising and wounding themselves against it.
— The Two Towers - Flotsam and Jetsam
He would have felt safe (and was in fact safe as the Ents had to besiege the tower).
He underestimated the anger of the Ents. He (his Orcs) had been cutting down vast swathes of Fangorn Forest down with no reprisal, and, had Merry and Pippin not met Treebeard he may have been safe in this assumption.
Pippin also alludes to this:
"He seems at one time to have got round them, but never again. And anyway he did not understand them; and he made the great mistake of leaving them out of his calculations. He had no plan for them, and there was no time to make any, once they had set to work.
— The Two Towers - Flotsam and Jetsam
- He thought that the Ents had all become more 'Treeish' to the point that they would not be able to do anything anyway.