I remember quite a bit of the manga, just not the name. So at the start, he dies and meets a goddess who give him a power like a skill tree. Then he wakes up in someone else's body, in a pool of blood. He realises that this person must have died. He looks at the skill tree and starts upgrading a few skills that he can. Then he gets attacked by, I think, centipede monsters and kills them. He comes back to the village and finds out he is an outcast from the village, and he referred to as the devil. The 3 guys that killed him the first time try it again, and get the crap beaten out of them.
Sounds like The Reincarnated Inferior Magic Swordsman
A normal office worker, Toru Minasuki, was swallowed up by a dimensional fissure and reincarnated into a different world. Upon reincarnation, Toru was given a [skill board] by God. Make the most of it to acquire all the necessary skills! Even though he is an “inferior person,” he’ll make his way to the top!
The part you describe fits perfectly with the first chapter. He is a office worker who falls through a dimensional hole and meet a goddess, who gives him a skill board.
When he wakes up, he finds himself in a cave, with blood coming from his head.
When leaving the cave, in chapter 2-3, he goes to the village, where everyone calls him "The Devil". After that, he is attacked by the three men you talked about, in the end of chapter 3 and beginning of chapter 4.
He beats them in chapter 5, and find the main girl, then leaves the village.