The first appearance of the word "Sith" in science fiction I have found is from Warlords of Mars by Edgar Rice Burroughs (1913-1914), which appears in these pages.
It seems to be a large insect:
Sith: A giant, venomous hornet-like insect endemic to the Kaolian Forest.
Porges, Irwin; Burroughs, Hulbert; Bradbury, Ray (1975). Edgar Rice Burroughs: The Man who Created Tarzan (1st ed.). Provo, Utah: Brigham Young University Press. p. 163. ISBN 978-0-8425-0079-1.
Burroughs may have derived the name from the Gaelic sith an alternative spelling of sidhe meaning fairy.
The next appearance is of a robot called Sith Tal a robotic museum curator:
From Amazing Adventures, Volume 2, Monsters of Living Flame (1951)
In his book The Secret History of Star Wars, Michael Kaminski suggests that Lucas likely got the word "Sith" from Edgar Rice Burroughs' John Carter of Mars series. In Appendix C of the book, Kaminski writes:
Lucas took many of his inspiration for names from other languages of the world, mixing and matching sounds and vowels and inventing his own to create a uniquely alien language that also seemed bizarrely plausible. ... "Sith" is a word Edgar Rice Burroughs invented, as is "banths" ("banthas")
Some other likely inspirations from Burroughs can be seen on this page (and also see this answer which links to a New York Times interview in which Lucas seems to confirm that the word "Jedi" was inspired by the word "Jed" for Martian Lords in Burroughs). Kaminski notes that when writing the original Star Wars, Lucas did a lot of research into old science fiction and fantasy works, including Burroughs, and quotes pages 141-142 of Dale Pollock's 1983 book Skywalking: The Life and Films of George Lucas which says:
Lucas returned from the local newsstand each weekend with a large collection of science-fiction magazines and comic books. Marcia wondered what was going on, but George told her not to worry, he was making a movie that ten-year-old boys would love...he thoroughly researched the science-fiction field from Buck Rogers and Flash Gordon to Stanley Kubrick's watershed film 2001: A Space Odyssey, made in 1968...Lucas also borrowed liberally from the Flash Gordon serials he had watched as a child, transplanting video screens, medieval costumes, art deco sets, and blaster guns to Star Wars...Lucas used Ming, the evil ruler of Mongo in the Flash Gordon books, as another model for his emperor. Alex Raymond's Iron Men of Mongo describes a five-foot-tall metal man of dusky copper who is trained to speak in polite phrases. From John Carter on Mars came banthas, beasts of burden in Star Wars; Lucas also incorperated into his early screenplay drafts huge flying birds described by Edgar Rice Burroughs. George watched scores of old films, from Forbidden Planet to The Day the World Ended, and read contemporary sci-fi novels like Dune by Frank Herbert and E.E. "Doc" Smith's Lensmen saga.
As Richard notes in a comment, the word "Sith" does appear in some early script drafts from the period when we would have been doing that research, like this draft from 1974 (other outlines and drafts including the word are mentioned in Kaminski's book as well).
In Chapter 2 Kaminski also quotes the first issue of the official Star Wars newsletter which again shows Lucas taking inspiration from Burroughs:
According to the first issue of Bantha Tracks, the official Star Wars fan club newsletter, Lucas also researched where Alex Raymond, the author and illustrator of the 1930s Flash Gordon comic strip, got his inspiration from, which led him to the John Carter of Mars novels by Edgar Rice Burroughs, which were similar in style and design.
And p. 16 of The Making of Star Wars provides what looks like a contemporary quote from Lucas about Burroughs as an influence:
"I've done a lot of reading for this picture. It's not really research so much as mythology and fantasy are taking over my life. I read everything from John Carter of Mars to The Golden Bough, so obviously all of that influences you in a certain way. I'm trying to make a classic genre picture, a classic space opera—and there are certain concepts that have been developed by writers, primarily Edgar Rice Burroughs, that are traditional, and you keep those traditional aspects about the project."
