Republic Era
The following states that :
Over the centuries, the Jedi amassed an almost exclusive right to the crystals, protecting and collecting them to decorate their temples and power their lightsabers. Some saw this collection of the kybers as proof of the Jedi's inordinate greed, and their secrecy over its nature a result of their wish to remain in power. Others saw the Jedi as protecting the galaxy from those who might use the powers of the crystal for dark purposes.
So you can say that before the rise of the empire, Kyber crystals were almost only used by Jedi, they controlled the planets were such minerals could be found and wouldn't allow the possession to anyone else.
Furthermore :
During ancient wars between the Jedi and the Sith, in which both sides fought for control over the galaxy, superweapons were created by the Sith that were powered by giant kyber crystals.
Even if the belief of the last Sith being killed long ago was spread among the Jedi, they couldn't possibly let any dark users of the force the opportunity to recreate such weapons.
Empire Era
The reason of the value of a Kyber crystal during the reign of Palpatine is easy to understand. After destroying the Jedi Order and melting all the lightsabers of the Fallen protectors of the Republic, the Emperor decreed that the possession, trade and use of these minerals and lightsabers was to be forbidden. Why ? First of all to get in the way of a potential new light side user, no Crystal, no lighstaber, no Jedi. Second is because the Death Star itself needed Kyber crystals to power its laser.
So that is why in both the Republic and the Empire these Kyber crystals were rare.
But why were they expensive ?
It seems to me that simply the rarity of a Kyber crystal and the thrill to add such an illegal item in your collection would be enough to increase its value.