Luke's thoughts were dwelling on his friends, Han and Leia. He inadvertently labeled her as his twin sister in his mind just as Vader was scanning him. Note that Vader knows enough about her identity that he's confident that he'll be able to find her and turn her to the dark side. This implies that he knows her identity.
“Give yourself to the dark side,” Vader urged. “It is the only way you can save your friends.”*
Luke closed his eyes. I’m sorry, Leia and Han. I’d do anything to save you, but I must resist the dark side. Suddenly, Luke felt a dull ache in his head, and sensed that Vader was using the Force to probe his mind.
“Yes, your thoughts betray you,” Vader spoke, confirming Luke’s suspicion. “Your feelings for them are strong. Especially for …”
Luke tried to block his thoughts — and failed.
“Sister!” Vader said. “So … you have a twin sister. Your feelings have now betrayed her, too. Obi-Wan was wise to hide her from me. Now his failure is complete. If you will not turn to the dark side, then perhaps she will.”
“No!” Luke screamed in anger as he ignited his lightsaber and rushed at Vader. Sparks flew as they traded blows in the cramped area, and Luke felt the hatred within him build with each passing second. You’ll never take Leia, and you’ll never take me!
Return of the Jedi: Junior Novelisation
Vader's mental powers with the Force are capable of invading Luke's mind. Again, the implication is that his secrets are laid bare
Luke withdrew further into shadow. He tried to hide, but there was no way to hide what was in his mind—Leia was in pain. Her agony cried to him now, and his spirit cried with her. He tried to shut it out, to shut it up, but the cry was loud, and he couldn’t stifle it, couldn’t leave it alone, had to cradle it openly, to give it solace.
Vader’s consciousness invaded that private place.
“No!” screamed Luke.
Vader was incredulous. “Sister? Sister!” he bellowed. “Your feelings have now betrayed her, too … Twins!” he roared triumphantly. “Obi-Wan was wise to hide her, but now his failure is complete.” His smile was clear to Luke, through the mask, through the shadows, through all the realms of Darkness. “If you will not turn to the dark side, perhaps she will.”
Star Wars: Return of the Jedi - Official Novelisation
Interestingly, the Marvel comic (Star Wars: Return of the Jedi 4: The Final Duel, billed as the "official comic adaptation") is very much clearer on the subject. Leia is mentioned by name.
Moving waaaay down the canon scale, the Star Wars: Return of the Jedi Storybook is even more explicit about what Vader knows.
Vader picked up the weapon. "Give yourself to the dark side, Luke' he
said. "It is the only way you can save your friends. I know your
thoughts. Your feelings for them are strong, especially for . . .
Leia! You love her. Obi-Wan was wise to hide her. If you will not turn
to the dark side, perhaps she will." He knew his son's emotions. He
knew exactly how to stir Luke's anger and fear.