Did Han know in the timeline of the Original Trilogy that Darth Vader was Anakin Skywalker and subsequently Leia's father? It would be better if Han Solo knows his Father-in-law.
There's absolutely nothing in the Original Trilogy films (nor their novelisations) to indicate that Han learned of Vader's identity.
Luke refers to Vader as his father in his coversation with Leia, but there's nothing to show that Han was subsequently told this secret, at least prior to the end of the film.
“Oh, no. No.” She laughed, “it’s not like that at all—Luke is my brother.” Han was successively stunned, embarrassed, and elated. This made everything fine, just fine.
He took her in his arms, embraced her, lowered her back down into the ferns … and being extra careful of her wounded arm, lay down there beside her, under the waning glow of the burning Star.
I'd imagine if she'd already told him that Vader was Luke's father, his thought process would have been more like...
“Oh, no. No.” She laughed, “it’s not like that at all—Luke is my brother.” "Bloody hell," thought Han. "But surely that means that Darth Vader was her father too!"
That all being said, Han evidently did find out some time in the intervening 20 years between Return of the Jedi and The Force Awakens;
What could he say to that? What possible response could he give to a willful denial of reason? “He’s gone, Leia. He was always drawn to the dark side. There was nothing we could’ve done to stop it, no matter how hard we tried.” His final words were the hardest to get out. “There was too much Vader in him.”
It depends on how you define "timeline of the Original Trilogy" - if you mean only what is shown on the screen, then Skooba and Valorum's answers are correct.
If you take timeline a bit more broadly, and from an in-universe perspective, just a single day after the end of Return of the Jedi, then the answer is:
Yes - Han knew.
From the new canon novel Bloodline:
Their romance had been so new then. Only the day before, he had offered to step aside if she wanted Luke instead. Would he be even quicker to leave once he knew she was a part of Darth Vader himself? But Han hadn't even flinched once. He had simply rocked her in his arms, giving her what comfort he could.
Bloodline - Kindle version (Chapter 23)
Han did not find out about the Vader/Anakin connection before the end of Return of the Jedi.
Luke had previously told Leia that Vader was his father, and then that she was his sister.
But when Leia tells Han she and Luke are siblings, she does not go further and reveal who their father is.
Han was either told later or was able to connect the dots prior to the events in The Force Awakens. Han says this while discussing his and Leia's son:
"He has too much Vader in him."
The (Non Cannon) Lego "Star Wars Droid Tales" shows Han Finding out. After the battle of Endor. When R2D2 gives C3PO his memory back.
Luke -Then you two fought in the Clone Wars with Obi-Wan?
C3PO - Indeed. And with Master Yoda and your father, before he became Darth Vader, of course.
Han - Wait, what? Luke's father is Darth Vader?
Luke - Everybody knows that. We learned of it months ago.
Han - I was frozen in carbonite. Sue me!