I read this book in the 1970s.
The central character comes in and out of a world where the color of the sun reflects the climate/chaos. The character enters the world when he experiences an epileptic (?) fit.
I suggest this phenomenon is the Sunbane, from the Second Chronicles of Thomas Covenant, a trilogy which came out in 1980-1983 (The first trilogy came out in the late 1970's). Every couple of days, the sky turns one of four different colors, making the sun appear of that shade. Each color is associated with a particular dramatic shift in climate. The Wikipedia article for The Wounded Land, the first of this trilogy, says
The Land is afflicted with the Sunbane, a disruption of the physical order which alternately causes rain, desert, pestilence and unnatural fertility to wreak havoc on man, animals and nature.
Thomas Covenant does not have epilepsy. He enters the Land when he suffers physical trauma (a blow to the head, an auto accident, etc.).
I'm forcefully reminded of Whipping Star and The Dosadi Experiment.
The genre is definitely Sci-fi rather than Fantasy, with the setting being an interstellar empire of sorts.
The timeline fits, Whipping Star came out in 1970, Dosadi Experiment in '77.
Travel and particularly communication via the Calibans' jumpdoors closely resemble an epileptic fit. Dosadi Experiment more-or-less starts with the protagonist having a communication fit.
What I can't match is the weather/colour link but I don't remember much about Whipping Star; the sun-climate link is made quite strongly at the beginning of Dosadi Experiment but only once.
I would also expect that the big talking toads would have made more of an impact on the memory than is apparent from the question.