I've been looking for a very similar book for a while now, and this question was the only thing that came up. I managed to track down the book I was looking for on my bookshelf.
It's called Tides of God, by Ted Reynolds.
From a Goodreads review:
In the 33rd century, after another round of Dark Ages, humanity has finally emerged with a utopian society built on the use of reason. Our more advanced alien allies loan us our first starship, on the condition that the crew hunt down and destroy a common enemy. The aliens tell us that this mysterious enemy is that long-forgotten entity, God. Twice before, it has passed Earth, each time driving us down into centuries of irrationality and bloody religious fanaticism, and it's headed back again. Now, if this were a James Morrow novel, that alien would actually be God. But in this, it is an unknown force that projects psychological belief in itself as God. Having no direct weapons, the projected madness is its only defense, as it causes the crew to turn on itself violently. Other than this one major innovative concept, the novel is a fairly routine space opera, with plenty of action and interpersonal intrigue. Somewhat more troubling, the author seems to have simply aligned several concepts without really examining their interrelationships, along the lines of reason=athiesm=libertarian sexual mores. At one point, I found myself cheering on the "breakdown" of the "rational" system of attachment-free sex. So while somewhat interesting in its portrayal of two extremes of human society, this book does not deserve the powerful acclaim implied by its cover blurbs.