Roger Corman's 1958 sci-fi film "Teenage Caveman" ( AKA "Out of the Darkness" and "Prehistoric World") is the earliest such story that I know of, though I doubt not that there are others.
Wikipedia synopsis: A tribe of primitive humans lives in a barren, rocky wasteland and struggle for survival, despite a lush, plant-filled land on the other side of a nearby river. They refuse to cross the river because of a law that evolved from an ancient tale warning of a god lurking there who brings death with a single touch. A young man of the tribe challenges the law and is eventually followed by other male members of his tribe, who fearfully cross the river in order to bring him back. They soon encounter the terrible god, a large, horribly burned but strangely human-like creature. Despite the young man's peace overture to the god, another tribal member, out of fear, lays a trap and stones the creature to death with a large rock; the young man then shoots and kills that tribesman with one of his arrows. The others gather around the now-dead god and discover that the creature is actually a much older man with long white hair. He is wearing some kind of strange, unknown outer garment with a fearful hood. They find another strange thing in the old man's possession; they are puzzled by this flat, thick object that opens and contains mysterious markings and vivid black, white and gray images that show an even stranger human world unknown to them. In a surprising denouement provided by the old man after his death, the truth is revealed in voice-over as the tribesmen page through his book: He was actually a survivor of a long-ago nuclear holocaust, forced to live for decades inside his now-ragged, discolored and bulky radiation suit (which is implied to have once been covered with deadly radioactive fallout). The old man has wandered the land for decades, while the primitive remnants of a devastated human race have slowly increased their numbers; his frightening outer appearance caused them to fear and shun him. A final, cautionary question is asked in voice-over by the old man: will humanity someday repeat its nuclear folly after civilization has once again risen to its former heights?
Jock London's post-apocalyptic story "The Scarlet Plague" (1912) also deals with a primitive culture following a civilized culture, but the characters, at the time-setting of the story (2073), are aware that they follow a more advanced society.