It's SF, not fantasy, but monks that do math, alternate "dimensions" (inter-universal travel) and multiple zones sounds a lot like Anathem (2008) by Neal Stephenson. It's definitely a tome; the trade and paperback editions ran over 1000 pages.
The story is set on Arbre where monastery-like places called "concents" are where the thinkers are cloistered.
Quoting the plot summary from Wikipedia:
The narrator and protagonist, Fraa Erasmas, is an avout at the Concent of Saunt Edhar. His primary teacher, Fraa Orolo, discovers that an alien spacecraft is orbiting Arbre – a fact that the Sæcular Power attempts to cover up. Orolo secretly observes the alien ship with a video camera, technology that is prohibited for the avout. Erasmas becomes aware of the content of Orolo's research after Orolo is banished (in a rite called Anathem) from the Mathic World for his possession and use of proscribed technology within the concent.
Fraa Orolo is banished (Anathem); Fraa Erasmus was called out (Voco) to attend the Convox called to deal with the alien visitation.
The alien ship actually has 4 populations of almost biologically compatible people from 4 separate universes with slightly different physical laws that render chemistry between their elements impossible; for example nuclei have different bonding energy:
...the aliens are found to come from planets in four parallel and distinct cosmos: Urnud, Tro, Laterre and Fthos. The multiple-worlds interpretation of the cosmos is discussed in great detail by the high-level avout at successive evening meals to which Erasmas performs the duties of a servant. In this section of the novel, it slowly becomes plain that Laterre is our own Earth, which serves as a 'higher plane of existence' for Urnud and Tro, and Arbre is itself a 'higher plane' for Laterre.
Even before getting physical samples, the avout deduce that the visitors have different physical properties because a laser used as a signal by the visitors is generated at a wavelength that can't be produced by any known element in Arbre's universe.
The "rings of the monastery" might correspond to the 4 "maths" of the concent: the Unarian, Decenarian, Centenarian and Millenarian which, respectively, communicate with the outside world every year, 10 years, 100 years and 1000 years.
The part about coordinates may be from the discussion of "Hemn spaces" which come up a lot in discussion about a coordinate system in which multiple universe lines can be defined and plotted.
The book is quite complex, especially the ending, and worthy of rereading.