Philip José Farmer is the first modern author that I know of to rework another author's scifi/fantasy character. He rewrote Edgar Rice Burroughs Tarzan character from a first person perspective in two novels: A Feast Unknown, 1969, and Lord of the trees, 1970. He then wrote the E.R.B character's fictional biography: Tarzan Alive: A Definitive Biography of Lord Greystoke, 1972. If P.J.F. isn't the first author to rework another author's storyline, who is, and what book or character was the story based on?
There are different versions of the Epic of Gilgamesh, so some unknown writer reworked an existing character. That's from around 1900 BC.
The Encyclopedia of Science Fiction has a great page on sequels by other hands. There were sequels to Gulliver's Travels starting in 1726, but that's too early. Within 250 years, their chronological listing says
Honoré de Balzac's Melmoth Reconcilé (1835; trans in coll The Unknown Masterpiece 1896) is such a sequel to Melmoth the Wanderer (1820) by Charles Maturin.
I think Conan the Barbarian fits the bill. Robert E. Howard created him in the 30's, but some of his stories were rewritten or completed by L. Sprague de Camp in the 50's, who also revised a lot of the setting. Then new non-Howard Conan stories were published in the late 60's/early 70's. I got most of my info from this Wikipedia page: http://en.wikipedia.org/wiki/Conan_(books)
Another possibility, even earlier, is H.P. Lovecraft's Cthulhu mythos, assuming you're also looking for shared worlds/mythos, not just specific characters. Lovecraft's stories were written in the beginning of the 20th century, but the main Lovecraftian mythos was defined by August Derleth and other authors after his death. But see @Pureferret's answer for more information.
In January 1926 HP Lovecraft wrote The Dream Quest of Unknown Kadath featuring on of his recurring characters Randolph Carter. He met Edgar Hoffman Price on June 12 of 1932, and by August Hoffman had written a sequel featuring Randolph called The Lord Of Illusions which has been subsequently published (1982) on it's on.
The purpose of writing this work was however to spur Lovecraft into writing another Carter story (or re-writing Price's story) as a sequel. In April 1933, Lovecraft reluctantly reworked it extensively (in the words of Price 'I estimated that [HPL] had left unchanged fewer than fifty of my original words')
My issue is that HPL started up something of a clique around the Cthulhu Mythos and I'm sure this wan't the only reworking of his characters, and there may have been an earlier/clearer example.
Information taken from my H.P Lovecraft anthology The Dreams in the Witch House, and Other Weird Stories.
How different does something have to be to count as a rework? Thompson's Oz books were billed as more of the same. However,
if the publishers had hoped for a writer who would produce near clones of Baum’s work, they were to be disappointed. Thompson may have used Baum’s setting, but she put an original stamp upon Oz right from her first page.
You can decide for yourself. Project Gutenberg has The Royal Book.
The War of the Worlds by H.G. Wells was serialized in 1897 and published as a book in 1898. Unauthorized and changed versions of War of the Worlds were published in the USA in 1897-1898 as Fighters From Mars. Earth struck back in Edison's Conquest of Mars (1898) by Garrett P. Serviss, a sequel to Fighters From Mars.
this is the first example in science fiction I can think of. Unless any earlier examples fit the definition well enough, it might be the first example.
(added May 11 2019. I may note that the fictitious anti gravity force apergy was first described in Percy Greg's Sword and planet science fiction novel Across the Zodiac (1880) and has been mentioned in various later stories. But that is more an example of writers reusing concepts and their names than writing sequels. https://en.wikipedia.org/wiki/Apergy 1)
The first continuations of Dickens' last and unfinished novel The Mystery of Edwin Drood were published in 1870, 1871-72, and 1873. If they fit the definition well enough, they may be the first examples.
I note that many medieval romances that have plot elements similar to modern fantasy novels like wizards and dragons and giants, etc., were continued in sequels.
For example the Romance of Amadis of Gaula is first mentioned in about 1350 and was possibly first written by Prince Henry of Castile (died 1308) but the first known edition was published in 1508 by Garcia Rodriguez de Montalvo who published a sequel, Esplandian in 1510. Fifteen further continuations or sequels were published by various writers in Spanish, Italian, and German by 1595.
The poems in the Ancient Greek Epic Cycle about the Trojan war included in order of legendary events:
Cypria attributed to Stasinus.
Illiad attributed to Homer.
Aethiopis attributed to Arctinus.
Little Illiad attributed to Lesches.
Illiou Persis attributed to Arctinus.
Nostoi attributed to Agius or Eumelus.
Odyssey attributed to Homer.
Telegony attributed to Eugammon.
They are known only from summaries and a few quotations except for the Illiad and Odyssey. The Illiad has gods interacting with men and the Odyssey has cyclops and witches and monsters, so the Epic cycle can be considered a fantasy series.
Obviously if the poems were written by different writers at different times some of them could be considered sequels to works by different writers.