All I can remember is that it's an isekai where his charm stat is tens of thousands in the negatives, but at some point it gets so low that it turns positive
It looks like you've already discounted this in the comments, but I Became a Living Cheat (Riaru de Reberu Age Shitara Hobo Chītona Jinsei ni Natta) really is a good match other than not being an actual isekai in that the protagonist is not transported to another world.
If a man ‘Levels Up’, any boring work can be repeated endlessly―Natsuno Taiyou.
“Talent” should be useful only in game, but even in reality how is it useful!?
When Taiyou woke up he received a power from a fairy “The ability to Level Up in the real world”, but immediately when leveling up boring work, surprisingly ‘charm’ had dropped down!―However, the charm level that was originally 0, has thank god become ‘65535′!
Taiyou aims to level 100. And enjoy a harem life with seven brides.