In the books of A Song of Ice and Fire as well as in the HBO adaptation, A Game of Thrones, whenever the white walkers make an appearance, it suddenly becomes really chilly and cold and everything turns into ice around them (water, trees, etc.). Are they responsible for the low temperature occurrences? Or is it the other way around meaning, that they can only appear when it is already very cold, which would kind of explain why they only appear during the mighty winter.
In the book A Feast for Crows; Jon Snow and Sam Tarly are talking about 'The Others' (The White Walkers from the show) after Sam has spent some time studying them in the library of Castle Black. This is what he says regarding this topic:
“Long ago,” Jon broke in. “What about the Others?”
“I found mention of dragonglass. The children of the forest used to give the Night’s Watch a hundred obsidian daggers every year, during the Age of Heroes. The Others come when it is cold, most of the tales agree. Or else it gets cold when they come. Sometimes they appear during snowstorms and melt away when the skies clear. They hide from the light of the sun and emerge by night... or else night falls when they emerge.
-A Song of Ice and Fire: A Feast for Crows, Chapter Five (Samwell I).
We don't know for sure, but there is a mention on the World of Ice and Fire concerning the Long Night, where it is mentioned that the Others brought the cold and the dark.
According to these tales, they [The Others] came from the frozen Land of Always Winter, bringing the cold and darkness with them as they sought to extinguish all light and warmth
But, those are tales that are hard to credit, so the Maesters can't be sure if it is the Others that bring the cold.
The TV show actually does give a definite answer to your question in season 5 episode 8: "Hardhome".
When Jon and the wildling Loboda search for the dragonglass daggers in the burning townhall, they encounter a White Walker warrior. The White Walker warrior walks through the fires without paying attention to them and the fires in his vicinity diminish and die out.
There is a clip on YouTube showing what I described here. So it seems that, in the TV show at least,
the White Walkers are responsible for the cold.
As per show canon the White Walkers bring the cold with them. This has been explained by the visual effects company that did the work for the Season 2 finale when the White Walkers and their army of wights are marching on the Fist of the First Men.
“On top of that we had to add the zombie makeup to the horse, and our actor had a bit of a belly, so we had to reduce his belly a bit, and we added some cold emitting from his body, so there’s a particle system coming from him. Of course we had to create his blue eyes, and a snow particle pattern on top of everything. We also replaced the green screen with a full CG environment and matte paintings and a dramatic sky and, of course, an atmosphere. The atmosphere was meant to make the shot look more beautiful and fit in with this idea that the White Walkers are bringing the cold with them. At the same time it was meant to hide from all the imperfections that we would get from all of these plates.
Fast Company, Collaboration Isn’t Always Pretty–Behind The Emmy-Winning VFX For “Game Of Thrones”