In the book, several characters do accept milk of the poppy. No less a person than Eddard Stark takes it, after he's attacked by Jaime Lannister in King's Landing towards the middle of A Game of Thrones.
Somewhere the gold cloaks found a litter, but the trip back to the castle was a blur of agony, and Ned lost consciousness more than once. He remembered seeing the Red Keep looming ahead of him in the first grey light of dawn. The rain had darkened the pale pink stone of the massive walls to the color of blood.
Then Grand Maester Pycelle was looming over him, holding a cup, whispering, "Drink, my lord. Here. The milk of the poppy, for your pain." He remembered swallowing, and Pycelle was telling someone to heat the wine to boiling and fetch him clean silk, and that was the last he knew.
A Song of Ice and Fire, book 1: A Game of Thrones, chapter 35
I do remember several times in the books when characters refuse it, often because they need to have their heads clear for some reason. I think the show creates the perception that everyone is always turning it down because when people take it in the books, it's usually mentioned in the narration; in the show, that all gets cut because it's not that important to the plot.