Fall of Angels, by L.E. Modesitt, Jr.
Your memory is pretty good, although you're giving Nylan short shrift - that's okay, because the point of the book is partially that Nylan, as a man, gets shorter shrift than he deserves.
To shamelessly quote the author's web page:
Modesitt moves deep into Recluce’s past to chronicle the founding of
the Empire of the Legend, the almost mythological domain ruled by
woman warriors on the highland plateau of the continent of Candar. He
tells the story from the point of view of Nylan, the engineer and
builder whose job it is to raise a great tower on the plateau known as
the Roof of the World. Here the exiled women warriors will live and
survive to fulfill their destiny. Here a revolutionary new society
will be born
. . . if Nylan can get the tower built and defenses in place before
the rulers of the lowland nations come with their armies to obliterate
them all. And if Nylan can learn to control the magical powers that
are growing within him.
It's the 6th book set in Recluce, albeit the earliest chronologically - it's the flashback to the foundational event that sets the stage upon which all the other books play. It's the arrival of Order magic on a world of Chaos magic.
