The author said that he is taking a break from the series right now, but that Leviathan is not the end.
After eleven books in The Lost Fleet and Beyond The Frontier in the last ten years, Leviathan marks not an end, but a pause point. I need time to figure out where to go next, how far forward to take the story, and what to explore that will involve new adventures and new challenges. (Besides, Black Jack really deserves a break.)
jack-campbell.com
In the interim, he will be writing The Genesis Fleet. From this I speculate that we will not see The Lost Fleet: Beyond the Frontier - Book 6 until this prequel trilogy is completed. So roughly 3 years, given his past rate of publication.
While I'm working on that, I'll be writing a new trilogy set in the Lost Fleet universe. The Genesis Fleet will be set centuries before the events in The Lost Fleet, showing the events, challenges, and people who led a bunch of fiercely independent star systems to first form the Alliance, as well as the first clashes between the Alliance and the expanding Syndicate Worlds.
jack-campbell.com