You may be thinking of "Herbig-Haro"; not a book, a novelette by Harry Turtledove; originally published in Analog Science Fiction/Science Fact, October 1984, reprinted in the 1986 anthology The Stars at War edited by John F. Carr and Jerry Pournelle.
"Herbig-Haro" was set about 12 centuries after the failed invasion of Earth by low-tech space aliens called the Roxolani, leading to the rise of the Terran Confederacy; or about six centuries after the collapse of that Confederacy. The Roxolani invasion itself is the subject of a prequel, "The Road Not Taken", first published in Analog Science Fiction/Science Fact, November 1985, which is the subject of a Wikipedia article and several answers on this site.
"The book takes place countless thousands of years later when the galaxy spanning Terran empire has collapsed millennia ago."
Maybe it wasn't quite galaxy-spanning, and maybe it's been a few centuries rather than countless millennia, but it has collapsed:
" 'The Road Not Taken,' " B'kila murmured.
Back then, on Terra, they knew FTL travel was impossible forever. It was a rude shock when they found that a couple of simple experiments could have given them the key to contragrav and the hyperdrive three, four, even five centuries earlier."
"How did they miss them?" Chang asked.
"No idea—in hindsight they're obvious enough. What's that race that flew bronze ships because they couldn't smelt iron? And every species we know that reached what the old Terrans would have called a seventeenth-century technological level did what was needed—except us.
"But trying to explain contragrav and the hyperdrive skews an unsophisticated, developing physics out of shape. With attention focused on them, too, work on other things, like electricity and atomics, never gets started. And those have much broader applications—the others are only really good for moving things from here to there in a hurry."
With a chuckle, Chang said, "We must have seemed like angry gods when we finally got the hyperdrive and burst off Terra. Radar, radio, computers, fission and fusion—no wonder we spent the next two hundred years conquering."
"No wonder at all," B'kila agreed soberly. "But the Confederacy grew too fast and got too big to administer, even with all the technology we had. And unity didn't last forever. None of our neighbors could hurt us, but we did a fine job on ourselves. Someone back then wrote that it was only sporting for humans to fight humans; no one else gave any competition."
"And so, the Collapse," Chang said.
"And here we are, on Loki and a few other worlds, picking over the pieces, a scrap from here, fragment from there, and one day we'll have the puzzle together again—or maybe a new shape, better than the one before . . . if we get the time. But those four missing ships frighten me."
P.S. From the Harry Turtledove Wikia:
"Herbig-Haro" is a short story by Harry Turtledove, published under his pen name Eric Iverson. It was originally published in Analog, October, 1984, and reprinted in Imperial Stars I: The Stars at War, edited by Jerry Pournelle & John F. Carr, Baen 1986. While set in the same universe as "The Road Not Taken", it has not been reprinted.
After the failed Roxolan invasion, Earth gained knowledge of hyperdrive and contragrav. With other technology far in advance of other species, humanity quickly expanded to other stars conquering the native populations and forming an empire called the Terran Confederacy. However, this expansion was too quick and too extensive. It became impossible to administer and so collapsed.
Some 1200 years after the first contact with an alien species, a few human colonies, including Loki, had regained most of the technological knowledge of the Terran Confederacy, both by rediscovery and by scavenging ruins. The Loki Salvage Service was created to continue this work and ca. AD 3240 it encountered an alien species on the other side of the Orion Nebula which had also taken "the road less traveled".