Unfortunately, it's not clear whether Sammael believe he had a truce or not. There is evidence to support both sides. I think, however, that the general consensus is that he was bluffing.
The Setup
Sammael sent a messenger to Rand, asking him for a truce. The terms of the truce basically keep Sammael confined to the southwestern part of the continent (The Illian / Altara region). He also agrees not to help any other Forsaken Rand might be going after. Rand, of course, refuses the truce, and the messenger dies a horrible, bloody death.
It turns out that Sammael didn't expect the messenger to actually return. Rather, he's set a weave on the poor guy that will cause him to die as soon as Rand gives his answer. The manner of his death will tell Sammael how Rand responded.
Later, Sammael claims (to Graendel) that Rand agreed, and that he has a truce, and thus free run of the southwest.
Option 1: He's Bluffing
The most obvious answer is that he's bluffing. It seems unlikely that he would have left room for misinterpretation in his weave on the messenger. So he should have known what Rand's answer was, and is just lying.
The point of lying here would be to make Graendel second-guess herself, and any plans she might have to attack Sammael. Remember, at this point, Moridin was not around yet, and the Forsaken were competing amongst themselves to be named Nae'blis. In addition to each of them trying to be the one that received that honor, they were also competing to curry favor with anyone who might win instead. Having brought Rand under control would be a huge feather in Sammael's cap, and almost certainly get him named Nae'blis. Knowing that, Graendel would be more likely to cooperate with him and avoid stabbing him in the back.
Beyond that, there was an implied threat to Graendel if Sammael is telling the truth: he had vowed not to help the others if Rand came after them. The unspoken threat here is that Sammael would "accidentally" let slip to Rand's people how to find Graendel, so he could get her out of the way. Staying on his good side would also serve to keep herself safe in the process.
One final benefit of tricking Graendel is that, if he makes her sufficiently paranoid, she might try to off Rand herself. One thing all of the Forsaken are
seriously worried about is that, this time around, Rand will join the Dark One and be named Nae'Blis himself. This is what the Dark One wants more than anything else -- to coopt the Dragon to his side. By telling Graendel that Rand has already agreed to cooperate with one of the Forsaken, there's the chance she might panic and kill Rand herself, thus removing Rand and herself from contention for Nae'Blis.
Besides making the most sense, it also fits with Sammael's behavior later in that scene, after Graendel leaves. As long as she's there, he's putting on a very confident front. Graendel is completely convinced that he's telling the truth. But as soon as she leaves, we switch perspectives to his inner monologues, and he seems much more smug about "handling Graendel" than he does about successfully making a truce. The implication is that his whole story was a lie, and he's smug that Graendel bought it.
(Unlikely) Option Two: He Was Set Up
There is an alternative explanation: Sammael actually does think he has a truce. Again, if we assume he's not an idiot, it's unlikely that he would make that mistake on his own. But that doesn't mean he wasn't set up by another male channeler somewhere along the way. It's a stretch, but not impossible that someone tampered with the weave, or layered a second one on top. It would require someone who could not only channel, but also somehow be aware of Sammael's messenger trick, but there's plenty of chances for another Forsaken to intercept the messenger.
There's some very thin evidence in this direction. First, Graendel seems pretty convinced that Sammael isn't capable of that kind of deception. It's the main reason she believes him in the first place. Also, later on Sammael learns that Mat is in Ebou Dar, he feels "betrayed". Ebou Dar is the capital of Altara, one of the areas Sammael claimed for himself in his proposed truce. It's possible that his betrayal is that Rand sent Mat into his territory, in violation of their truce.
Of course, there are plenty of other ways to interpret that "betrayed" feeling, so it's not very compelling evidence. But we also don't get enough information to rule it out. Fairly shortly, Moridin arrives and is named Nae'Blis and all the in-fighting among Forsaken comes to an end, and the whole thing is dropped.
