This is what I remember from the manga. The manga starts out with our main character getting kicked out of the party he is in (I think it is the hero's party) for being to weak. It turns out that he has given each member of the party some of his power over the last 3 to 5 years he has been with them. While leaving he takes his powers from them and because they weren't giving him anything back it prompts him to take some of their power to make up for the massive debt of skills they have accumulated. He ends up taking a lot of their power because of the amount of time they were using his power he had given to them. The amount of debt that accumulated was in the millions.