This is what I do remember from the manga: in the beginning it turns out that our main character was summoned from another world but was either kicked out of the castle or was able to leave because they say he had no overpowered skill.
His skill actually turns out being able to manipulate skills and fuse them together to create new skills. He does this and sells the skill to then buy a slave that these spirits told him to buy because the spirits wanted him to save her.
After buying her, the person he sold the skill to then thought he might have more skills to sell him or a way to obtain those skills, so they go after our main character.