No.
Steve Rogers is not a virgin. He lost his virginity to a girl in 1943 on the USO tour.
Bruce Banner, Steve Rogers' co-Avenger and friend, reveals this to Jennifer Walters (She-Hulk) in the mid-credits scene of She-Hulk: Attorney at Law S01E01, "A Normal Amount of Rage":
Jennifer: [Drunk crying] It's just so sad. Steve Rogers did so much for his country and he never got to experience sex.
[Crying] Did you see that ass? Like, that ass did not [Cries] deserve to die a virgin. It's, like, so sad.
Bruce: Steve Rogers is not a virgin. He lost his virginity to a girl in 1943 on the USO tour.
Jennifer: Yes! I knew it.
Bruce: You're not drunk?
Jennifer: [Shouts] CAPTAIN AMERICA FU…!!! [Scene abruptly cuts off to credits]
The context to the above scene was that earlier in the episode, Jennifer was relating her theory to Bruce that Captain America was a virgin (probably referencing the real-life fan speculation and in keeping with the self-referential and fourth-wall-breaking nature of the show and She-Hulk).
Jennifer: Okay. So, back to what I was saying before, my theory.
Bruce: Oh, God.
Jennifer: Um, Steve Rogers did not have a girlfriend before he went into the service.
Bruce: Says who?
Jennifer: The History Channel. So, he becomes Captain America. And from that moment on, a symbol of America, he is rushed to the frontlines, he becomes a war hero. Then he is frozen in ice.
Bruce: Right. Right.
Jennifer: So, based on everything you've told me, after he gets unfrozen, he goes from world-threatening disaster to world-threatening disaster. That's when he's not a fugitive from the law, right? So it seems like he was pretty, pretty busy.
Bruce: All you're doing is repeating everything that I've already told you about my friend and colleague.
Jennifer: Obviously, Captain America was a virgin.