I have a strong memory of a scene in a time-travel book without remembering many other details.
The main character is trying to accomplish something (I don't recall what, but call it X). In a wooded valley, he comes across a guy wearing some kind of suit that makes him unrecognizable. The guy is repairing a timeship. The guy attacks him, and they have a fight. Later, the main character accomplishes X, but it turns out to be disastrous. He uses a timeship to go back and stop himself. So the fight is between him and himself from a different time stream. At first he thinks X is a good idea, but later realizes it isn't.
I imagine I read this in the 60's or early 70's, but it may have been written earlier. For a while I thought it was Time and Again, by Clifford Simak (it's not), and then I thought it was The Door Into Summer, by Robert Heinlein (also not). Both of these books have similar time-travel situations, but not the guy fighting with himself.