In short, we don't know. In the next film we may see inside the books and answer the question in a more definitive way, but it seems likely that we're going to find out that the books are written in Aurabesh, written Galactic Standard.
The Visual Dictionary states that Luke assembled this collection in his quest to explore the remains of the Jedi, implying that they were neither written at the same time, nor kept together.
Luke's travels have allowed him to collect and assemble ancient scriptures containing lost Jedi wisdom and abilities. Actual books such as these are a true primitive rarity.
Since the earliest Jedi lived (tens of?) thousands of years BBY, it stretches credulity that these are the literal earliest writings of the Jedi given their condition, differing book sizes and binding styles.
Given that these are likely to be copies of copies of copies (ad infinitum) and like many copies of sacred texts, more than likely to have been translated into a more accessible languages, and given that Aurabesh predates both the first Jedi and hyperspatial travel itself (at least within the Legends canon) there's really no good reason to assume that they weren't just written in plain, boring old Aurabesh or, if they were originally written in another less accessible language, subsequently translated into Aurabesh over the millennia.