I didn't see a reason for him to do this in the episode, I have probably missed something.
But, why did Phil Coulson leave his prosthetic hand behind on the alien planet?
This is never explicitly stated, either in the show or (as far as I've been able to find) by the showrunners.
However, my interpretation would be first in-universe that Coulson was feeling a great guilt for his actions both in that moment and leading up to it; choosing to follow his desire for revenge has left him feeling very conflicted. He may even feel that he has lost a part of himself and as a result may be wishing to attempt to move on from that by leaving the hand behind.
This is backed up by these quotes from the cast and crew, firstly from Clark Gregg himself:
"On the one hand, it means Coulson having crossed a line that I think he takes very seriously, which is, 'If you're trying to protect the world from evil and you then become evil, haven't you become part of the problem?' That's a little bit what's going on for him, even though I don't think that answer's cut-and-dry in this case."
And from Maurissa Tancharoen:
"He's been grappling with his humanity the whole time, and I think that act of doing the spoiler act [has an impact]. We will definitely see the emotional consequences of that in the back half of this season. I think the first natural thing for any human to do is live in denial for a while and just plug ahead. You may see that across the board."
I think from both of these you can see that he may be trying to move on from this part of his life, and leaving the hand behind may be his way of doing that.
There is also the idea that the hand is evidence of the crime, and Coulson may also not want that hanging around.
Finally, I think it is greatly symbolic for the end of a two-and-a-half season long story arc coming to a close, and the exit of the character who has been so central throughout. Losing this character could be seen as as much of a blow as Coulson losing his hand.