This is what I remember from the manga. The manga starts out with our main character getting kicked out of the party he is in (I think it is the hero's party) for being too weak. It turns out that he has given each member of the party some of his power over the last 3 to 5 years he has been with them. While leaving he takes his powers from them and because they weren't giving him anything back it prompts him to take some of their power to make up for the massive debt of skills they have accumulated. He ends up taking a lot of their power because of the amount of time they were using his power he had given to them. The amount of debt that accumulated was in the millions.
Marz was a member of the S-Rank party [Sword of Divine Silver]. Even though he had the unique skill [Skill Lender] that allowed him to lend skills to his party members, he was ostracized and taken advantage of because he did not fight on the front lines. Eventually, they tried to get him killed by the dungeon boss to avoid paying him his severance pay. Marz retrieved the skills that he lent to his party members with compound interest and instantly became the strongest. He then defeated the boss, obtained a 10/10 waifu and saved a demi-human slave to form a harem.
The protagonist, Marz, had the ability to lend skills to the other members of his party. Because he didn't fight on the front lines himself though, he was undervalued by the rest of his party, who took the skills he lent them for granted.
They decide to ditch him, but if they just fired him, guild federation rules would oblige them to pay him severance, so to avoid that, the leader of the party orders him to tackle the boss of a dungeon alone, hoping he'll be killed in the process.
Marz realises this, so he decides to take back the skills he lent the other members of his party before tackling the dungeon boss. And due to years of compound interest, he was entitled to seize tens of millions of skill points from them.