I doubt you'll find anything on it in the books.
I think this frost and cold imagery associated with Wild Hunt is the invention of the games - more precisely, it's an attempt to reconcile the more canonical image of Wild Hunt in the third game with how it was portrayed in first one, where...
...in the game's finale, Geralt is lunged into a vision of the post-White Frost future, where he battles the spectres of Wild Hunt amidst a landscape covered in snow and ice.
If you're looking for a more in-universe explanation, I would think it's part of the "lights and smoke" ensemble that Wild Hunt employs to appear more terrifying and otherworldly to the inhabitants of the worlds they raid.
I wouldn't try to rationalize it with "this is what you get when travelling between dimensions without a proper anti-frost coating on your plate armour" ;)