I read this book a 5-7 years ago. The parts I remember are:
He got his immortality from something his father did. And it was something so bad that he was banned from his home/galaxy.
He meets a woman along the way, and she describes his skin as being like sandpaper due to all the little scars he accumulated over his long lifespan.
When he travels for long periods of time in his spaceship, he covers all the clocks so he can't see the time.
That's all I can remember.