This is the 1974 story "Summer's Lease" by Joe Haldeman, originally published in Analog as "Truth to Tell". It is reprinted in Haldeman's collection Infinite Dreams. The story is exactly as you outline.
The planet is plagued by regular "Burnings", when every 80 years or so, one of the two stars the planet is orbiting flares up.
The story ends with the stranger getting paid off by the community leader.
...."How did it go?"
"He believed it, every word. You won't have any more trouble."
(p. 152, Infinite Dreams, 1978 St. Martin's hardcover edition)