"The Specter General", a novella by Theodore R. Cogswell, first published in Astounding Science Fiction, June 1952, available at the Internet Archive. Here is the plot description from the Wikipedia article about that story:
The 427th Light Maintenance Battalion of the Imperial Space Marines is sent to an uninhabited planet to set up a base, but is forgotten when the Empire collapses in civil wars. Generation after generation for 500 years, the soldiers stubbornly hold onto their mission, training to repair starships that never come. Their equipment gradually wears out and to all outward appearance, they seem to be primitive savages, living by hunting and farming. Periodic visits by the Inspector General (actually the battalion commander hidden in a still-working "battle suit") maintains the fiction that there is still a functional Empire.
A ramshackle Galactic Protectorate eventually emerges from the rubble of the Empire, though incessant power struggles keep it weak. Leaders strip subordinate commanders of trained personnel to strengthen their own positions while discouraging any ideas of rebellion. As a result, space fleets receive inadequate maintenance, and the level of civilization ebbs.
The current Lord Protector tries to purge the second most powerful man, General Carr, but he escapes. Pressure is put on sector and base commanders to find the fugitive. Conrad Krogson, head of War Base Three, takes his fleet out to search star systems that have not been visited in living memory.
Meanwhile, Colonel Marcus Harris, commander of the maintenance battalion is overthrown by his executive officer, Lieutenant Colonel Blick. He and a loyal Second Lieutenant Kurt Dixon are jailed. When Dixon escapes, he has to hide in a battle suit to avoid detection. He accidentally activates it and is propelled into outer space. There, he is captured by a scout and taken to Krogson.
Believing he has found Carr's base, Krogson prepares to destroy it. However, Dixon manages to rewire the circuits so that an attempt to fire would result in the fleet's destruction. A Mexican standoff ensues. Harris, restored to command to deal with the unprecedented situation, tries to negotiate with Krogson, but neither man is able to find a safe way out of their dilemma. Harris will not surrender his unit to Krogson, nor can he take a vastly larger force prisoner.
Then Krogson receives news that Carr has overthrown the Lord Protector and that he is now on the list of those to be purged. This breaks the deadlock. Harris proposes that his men whip the fleet into shape and make it an unbeatable force to take over the Protectorate. Krogson agrees, making it clear that he intends to set up a less cutthroat form of government. Then, Conrad Krogson, "Inspector General of the Imperial Space Marines", lands to inspect his troops.