It's remotely possible that you're thinking of The Elfstones of Shannara by Terry Brooks. The main character, Wil Ohmsford, is grandson to Shea Ohmsford, who bore the legendary Sword of Shannara from the book of that name, and was at that time the only living descendant of the Jerle Shannara, the Elven King who defeated the Warlock Lord. Long story short, Wil carries the Blue Elfstones, as were given to him by his grandfather, Shea, who received them from the Druids. He is apprenticed to be a Healer when the plot kicks in. However, he does not acquire another set of elfstones in the course of the books, and he lived in a village, not a castle. It's also a good bit older, although it's been reissued a few different times (and will probably get another release to tie into the TV show).
Of course, the Elfstones showed up in multiple books, so it could be any of a number of the sequels. In particular, the The Dark Legacy Of Shannara follows descendants Redden and Railing Ohmsford who eventually recover three more sets of elfstones.