Yes, it's in the books. Olenna did call Mace Tyrell "Lord oaf" and much more.
A Storm of Swords, chapter 6, Sansa I.
“Renly was brave and gentle, Grandmother,” said Margaery. “Father
liked him as well, and so did Loras.”
“Loras is young,” Lady Olenna said crisply, “and very good at knocking
men off horses with a stick. That does not make him wise. As to your
father, would that I’d been born a peasant woman with a big wooden
spoon, I might have been able to beat some sense into his fat head.”
“Mother,” Lady Alerie scolded.
“Hush, Alerie, don’t take that tone with me. And don’t call me Mother.
If I’d given birth to you, I’m sure I’d remember. I’m only to blame
for your husband, the lord oaf of Highgarden.”
“Grandmother,” Margaery said, “mind your words, or what will Sansa
think of us?”
“She might think we have some wits about us. One of us, at any rate.”
The old woman turned back to Sansa. “It’s treason, I warned them,
Robert has two sons, and Renly has an older brother, how can he
possibly have any claim to that ugly iron chair? Tut-tut, says my son,
don’t you want your sweetling to be queen? You Starks were kings once,
the Arryns and the Lannisters as well, and even the Baratheons through
the female line, but the Tyrells were no more than stewards until
Aegon the Dragon came along and cooked the rightful King of the Reach
on the Field of Fire. If truth be told, even our claim to Highgarden
is a bit dodgy, just as those dreadful Florents are always whining.
‘What does it matter?’ you ask, and of course it doesn’t, except to
oafs like my son. The thought that one day he may see his grandson
with his arse on the Iron Throne makes Mace puff up like . . . now,
what do you call it? Margaery, you’re clever, be a dear and tell your
poor old half-daft grandmother the name of that queer fish from the
Summer Isles that puffs up to ten times its own size when you poke
“They call them puff fish, Grandmother.”
“Of course they do. Summer Islanders have no imagination. My son ought
to take the puff fish for his sigil, if truth be told. He could put a
crown on it, the way the Baratheons do their stag, mayhap that would
make him happy. We should have stayed well out of all this bloody
foolishness if you ask me, but once the cow’s been milked there’s no
squirting the cream back up her udder. After Lord Puff Fish put that
crown on Renly’s head, we were into the pudding up to our knees, so
here we are to see things through. And what do you say to that,
And again in the same chapter.
As the servants brought out a broth of leeks and mushrooms,
Butterbumps began to juggle and Lady Olenna pushed herself forward to
rest her elbows on the table. “Do you know my son, Sansa? Lord Puff
Fish of Highgarden?”
“A great lord,” Sansa answered politely.
“A great oaf,” said the Queen of Thorns. “His father was an oaf as
well. My husband, the late Lord Luthor. Oh, I loved him well enough,
don’t mistake me. A kind man, and not unskilled in the bedchamber, but
an appalling oaf all the same. He managed to ride off a cliff whilst
hawking. They say he was looking up at the sky and paying no mind to
where his horse was taking him.
“And now my oaf son is doing the same, only he’s riding a lion instead
of a palfrey. It is easy to mount a lion and not so easy to get off, I
warned him, but he only chuckles. Should you ever have a son, Sansa,
beat him frequently so he learns to mind you. I only had the one boy
and I hardly beat him at all, so now he pays more heed to Butterbumps
than he does to me. A lion is not a lap cat, I told him, and he gives
me a ‘tut-tut-Mother.’ There is entirely too much tut-tutting in this
realm, if you ask me. All these kings would do a deal better if they
would put down their swords and listen to their mothers.”
But no he's not that foolish. Mace knows that the Lannisters need him, and tries to use this for the benefit of his house and for personal benefit as well. Maybe he is too ambitious and sometimes this blinds him, like when he approached Cersei during the funeral of Lord Tywin (AFFC, chapter 7, Cersei) and hinted that he wants the Hand's position, and also reminded her that Tywin did promise the position of Master of Coins to his Uncle. This angers Cersei and in the end no Tyrells make it in Cersei's small council. So Mace serving under Cersei happens only in the TV show.
George RR Martin thinks Mace from the TV show is portrayed like an idiot. He states that the character in the show combines Mace with Harys Swyft, and actually seems more like the latter.
Of course, it could also be a subtle bit of characterization, as you suggest, intended to show that Mace is an idiot who does not know his Westerosi history. (Not a mistake that Book Mace would make, but the character in the show combines Mace with Harys Swyft, and actually seems more like the latter).
Spoilers, short summary of future events:
After Cersei gets imprisoned by the Faith, she confesses her guilt and is forced to step down as Queen Regent. Tywin's brother Kevan starts to act as new Regent and Mace Tyrell gets his desired position, and becomes the new Hand of the King. How good a Hand he will make will be revealed after The Winds of Winter (the next and much awaited book from ASOIAF series) gets released.