I read this book probably close to 20 years ago (and even then I don't think the book was new). It might have even have been a short series.
It's in the future and the descendants of some spacefaring people have ended up on a planet and a small group of them have "magic powers." These powers are granted by symbiotic crystals that are tempered with various elements, e.g. fire/water. Depending on what type of crystal they bind with, it gives them control over the related element. I think this process of binding to the crystals happens in the spaceship that brought the original people to the planet.
I think that the main character binds with a crystal that wasn't tempered by any element and thus can control all of them?