I think I read this in Analog around 1980. A man in a pre-industrial type village believes that they originated on another world and keeps trying to persuade everybody of this. Nobody else agrees with this but they accept him as a harmless eccentric. Then one day he is visited by a stranger who tells him that he is absolutely correct - the village is a lost colony which has only recently been rediscovered, but the truth can't be revealed right away as they have to find the right way to break the news without causing panic. The stranger asks the man to stop trying to persuade people of his beliefs until the time is right, and the man agrees. At the end of the story it turns out that the stranger actually comes from another village on the far side of the mountains and was hired by the community leader to shut the man up.
This is the 1974 story "Summer's Lease" by Joe Haldeman, originally published in Analog as "Truth to Tell". It is reprinted in Haldeman's collection Infinite Dreams. The story is exactly as you outline.
The planet is plagued by regular "Burnings", when every 80 years or so, one of the two stars the planet is orbiting flares up.
The story ends with the stranger getting paid off by the community leader.
...."How did it go?"
"He believed it, every word. You won't have any more trouble."
(p. 152, Infinite Dreams, 1978 St. Martin's hardcover edition)