I am trying to remember a book I read a while ago. It is about a boy who lives in some sort of special mathematical conclave that only opens its doors to the outside world every few years. An alien spaceship is discovered orbiting earth and the boy, along with a few others, travel to space to take over the ship.
This sounds very much like Neal Stephenson's Anathem.
Special mathematical conclave:
In the novel, avout follow a life path called the Discipline, sometimes referred to as Cartasian Discipline, after Saunt Cartas, the founder of the mathic world. It is a set of rules governing what is (and is not) allowed for avout to know and/or do, and was codified centuries before the time of the story in the Second New Revised Book of Discipline.
Chief among these is that the avout are separated, both mentally and literally, from the Sæculum, or outside world.
The boy and the alien spaceship:
The narrator and protagonist, Erasmas, is a fraa at the Concent of Saunt Edhar (Saunt, abbreviated St., is a corruption of the ancient word savant and is a title bestowed on influential intellectuals of the past). His primary teacher, Orolo, discovers that an alien spacecraft is orbiting Arbre — a fact that the Sæcular Power attempts to cover up. [...] The avout team boards the ship [...]