It was mentioned first in "Grave Peril":
"Poor little poppet. All of your efforts to learn and you still know so little. Harry made a bargain with me long ago - and broke it, once then, and once a few nights past. He swore to uphold it again, last night, and broke it thrice. Now he reaps the consequences of his actions. His own powers turn against him, the poor dear, to encourage him to fulfill his word, to keep his promise."
So question is: when he broke his word, what were the effects on him? In Grave Peril, during Bianca's party, he basically became powerless when just meeting Lea there, but there must have been more. Bargains made there and later saw him get it back, but question is: since his first broken bargain with Lea was his power reduced and how?
Was it reducing his output (that is making him weaker)? Or was it siphoning some of his power to Lea? After all the oaths were on his power, so it's not unreasonable to think it was some sort of security on his oath (a lien of sorts)? Was it all the time or just in Nevernever, where Lea had power to exact the payment of the debt?
Or maybe it was just a reminder of his "treachery" - sort of like being pestered by bank: not really bad thing (yet), but annoying to the extreme?
When later Ms Sommerset paid him a visit his debt was a factor, yet it was our world, not Faerie... And still later, when he renegotiated, did his power - if he had parted with any - retuned to him?
As time passes Harry is getting better with magic, more precise, so it would definitely align with his recognition that he was more economical with it, but was it only that? Was being frugal with power (which is not surprising - he is part Scottish, after all) the only explanation for him getting better and stronger? Or maybe he got back what he lost (if he lost anything)?