First of all I would like to thank you for asking a really good question, it's been a long time since I thoroughly enjoyed writing an answer.
Since this question is about a certain aspect of Tyrion's personality, I think we need to discuss which events affected the growth of that certain aspect of his personality first.
Tyrion was more than aware of his short-comings. He was born in a family famed for beauty, wealth and strength. He lacked two of them and had wealth only thanks to his father. To make matters even worse, his father was a cold man who had an ideal about what Lannisters should be like, which was perfection. Anyone who didn't meet that ideal was beneath him. And sadly for Tyrion, he didn't quite manage to pass on that criteria.
For the outside world, Tyrion learnt long ago that while cruel words can hurt, it hurts even more to let people know that they got to you. So he was most accomplished at wearing a facade, a mask which didn't let slip that he was hurt by names such as imp etc and other cruel japes.
But that Facade never helped him when he was in front of his father. Whenever his father's piercing gaze was upon him, Tyrion suddenly became hyper-aware of all his short-coming. He could sense how his waddling was embarrassing, how his height was embarrassing, how his mismatched legs were embarrassing, how his mismatched eyes were embarrassing and how he was the very personification of Embarrassment.
That sort of up-bringing crushed his self-esteem but the kindness of his uncles, Aunt and his brother saw him through that and ensured that the poor lad didn't commit suicide. They however could not lift his own sense of self-worth (His uncles did try, for example Gerion Lannister used to make him recite 16 wonders of the world in front of entire West whenever there was a feast, when Tyrion was a child. Tyrion felt very proud of his accomplishments. His aunt Genna, who took up a maternal role for the Lannister orphans after Joanna died, also recognized Tyrion's intelligence and used to favor him instead of Cersei apparently), at least not around his father.
Tyrion's Love Life
With that done, now we have to discuss the cases of the Women he has been romantically involved with and those who want to be romantically involved with him.
Tyrion's first and for almost a certainty, only love, comes first. She's the only girl who ever truly and unconditionally loved Tyrion for what he was, Tyrion reciprocated with same if not deeper feelings.
She was an orphan, daughter of a Crofter, on her way to nowhere really as she lost her place when her father died. He met her on road while she was running from two men who were trying to rape her. Jaime chased the men and went back to the Rock to get more help as it offended him that Outlaws would dare strike on innocent travelers this close to the Rock. Tyrion on the other hand escorted Tysha to an inn, fed her and before they knew it, they were sharing a bed. Tyrion recalls that she was very lovely. She was a virgin and by morning, Tyrion was in love with her.
Tyrion married her by bribing a drunk septon. But he was very well aware that his father would never accept a Crofter's daughter as his daughter-in-law, wed to second in line to the throne of the ancient Kings of the Rock. So he found a way but it didn't work out very well. He set her up in a cottage and they played on being husband and wife for few weeks until the Septon he had bribed to marry them told Lord Tywin.
Then Tywin crushed him. He made Jaime confess that he had hired a virgin whore to make a man out of Tyrion. Then he gave her up to his garrison and bid Tyrion watch. Then he had Tyrion take her as well.
Tyrion hated his father for that. It crushed Tyrion, made him think that no normal woman would ever love a creature as hideous as him. It was all for his money, all a ploy.
He was still hurt by it, it's his POV, he thinks that smoke was stinging his eyes when he was retelling the tale to Bronn, when in fact was on verge of crying.
But it all made sense, right? He only had to look at himself. Why would a girl like Tysha want to be with a hideous malformed dwarf like him? He only had his money, yes that's what it was all about. His father was right. She was a gold-digger.
Except that she wasn't. She was just an innocent girl who truly loved Tyrion, who saved her from rapists, made her laugh, reassured her. Lord Tywin forced Jaime to confess that she was a whore, he told him that she only wanted Tyrion's gold so she was no better than a whore. And that Tyrion would thank him in future for getting him rid of this low-born whore.
This belief was cemented even further as he saw his father trying desperately to find a match for him among the nobles. He got so desperate by the end that he even offered to take soiled daughters of minor Lords as his daughter in law. And even they rejected the proposal.
Tyrion was now sure, why would any woman ever want him unless he paid her? So he started seeking solace among whores, as they were the only women who'd ever want to be with him, even if it was just for his gold.
But the ghost of Tysha never left him. It kept haunting him. He kept telling himself he was free of Tysha, yet he never was. Even as he was on brink of death in the aftermath of Battle of Blackwater, he was subconsciously dreaming about the brief happy married life he had with Tysha.
He was never free of her. He always loved her. Whatever hatred he felt towards her turned into enormous guilt as Jaime told him the truth. He had watched his wife getting raped and he actually took a part in the rape.
He went to his father to get the truth. When his father kept referring to her as a whore, he shot him. And now he goes around, wondering, Where do the whores go? So that he could find Tysha again1.
Shae is another one. While some part of Tyrion always kept telling him that you fool, she is just a whore, he couldn't help but fall for her.
Shae was however only interested in his money. And as soon as Tyrion was arrested on charges of regicide, she went to the camp of the highest bidder, which was Cersei. On promises of a wealth and a good marriage, Shae provided a false testimony in the trial, incriminating both Tyrion and his wife Sansa.
Tyrion was hurt but he knew what she was and he demanded a trial by combat instead.
Even though Tyrion thought the match was cruel, he agreed to it as he thought it was best to have Walls of Winterfell around him as he had Cersei for an enemy and his own power was dwindling day by day. Especially as Tywin made it clear to him that he would never let Tyrion inherit the Rock.
Tyrion discovered that he actually wanted Sansa, romantically. He wanted her to love him and he wanted to love her. But Sansa was always cold and distant. Tyrion thought if he was kind to her, she might come around. Everyone moves on, right?
That didn't happen. Sansa made it clear to Tyrion while she was willing to obey her Lord husband, she will never want a sexual relation with him out of her own will. That left a bitter taste in his mouth, further reaffirming his earlier conviction that he will never have someone who could love him for what he was. He told her as much, that that's why gods made whores for people like him.
Then there's case of Penny. Another lowborn dwarf girl. Tyrion thinks of her as both ugly and beautiful but by the time he met her, he was very much damaged goods. He was searching for Wherever the whores go, remember? The girl however was innocent and she began to like Tyrion.
Tyrion however failed to return the feelings as he didn't feel that way. But he was never cruel to her, he comforted her and rejected her advances by sympathetically lying that he was saving himself for his wife Sansa even though it's Tysha he is looking for. Lies we tell for love!
But then Shae came to haunt him. Young Penny expressed her sincere admiration for Tyrion, how he was brave. But that reminded him of Shae and her lies. Which enraged him.
Yet Tyrion doesn't hate her. He pities her rather, or feels sympathetic towards her.
Her raw innocent emotion and desire to find love reminds him of his own young self, when he was in love with Tysha. He doesn't want to crush that.
Penny is Tyrion's test. She is innocence, goodness and sincerity. Where as Tyrion now is hate, vengeance and indifference.
He doesn't want to break her heart but he can't find it in him to give her what she wants. He doesn't desire her sexually and he knows full well that she doesn't want him in a lustful way either, she's too innocent for lust yet.
Only time Tyrion was ever capable of loving someone was when he met Tysha. She was the only one and she has always been the one for him even though he kept telling himself otherwise for years until Jaime told him the truth.
Does Tyrion have such a low self-esteem?
Yes he does. He has had a rather dim view of his worth ever since he was an ugly child in Casterly Rock. If not for his uncles, Aunt and brother, he probably would have ended his life a lot earlier. He let himself believe that he could be loved when he met Tysha but he was reminded by his father that its not true. He kept believing it till Jaime told him the truth.
His teenage, his youth are formed based on that experience. He had accepted that as his fate. If its his fate to be accepted only by whores, then so be it.
Does he simply despise women of lower-classes?
No he doesn't. His only true love is low-born, the one who still haunts him and is driving him mad. He is capable of kindness to them, as he showed kindness to Robert's whores and Penny.
But why does he so quickly judge them guilty?
Take it case by case.
In case of Tysha, he just had to look at himself. Why would a beautiful girl like her want to be with him? Jaime was kind to him always, it was not out of character for him to hire a whore to make a man out of him. And why would his father lie about it? If he didn't like the marriage, he could just say that. Why would he want to lie? Tyrion had no reason to suspect that it wasn't true and the lie actually made sense to him.
In case of Shae, a woman who on numerous occasions reminded you that she was a whore indeed, lied in court to get you condemned to death presumably on behest of your sister with promises of gold, is now lying naked in your father's bed who just had condemned you to death, calling him My Lion, what other proof do you need to judge her guilty? Did Cersei Threaten to kill her? If she had been threatened, she could just provide a simple testimony, why add the colorful description of their intimate relationship? Why tell about their personal things in the court? And why was she now so happily lying in Tyrion's father's bed? She never cared about him.
In case of Sansa, she again proved he was not someone who could be loved, no matter how hard he tried, no matter what he did, no matter how kind he was. He fought with a King for her honor, fought with his father for her chastity but she wouldn't ever consider letting him touch her, she wouldn't ever confide in him, she wouldn't ever consider him her life partner. And when he was arrested, she was first to fled, betraying her husband. But still Tyrion to defended her in the court as innocent.
In Case of Penny, he is damaged goods. He doesn't desire her. He doesn't want to tarnish her innocence either. But pities her, he sees himself in her but he doesn't love her. Nor does he hate her and would do what he can do to protect her. Goes without saying she is lower class.
1. There is a fairly convincing theory that Sailor's Wife, a whore in Braavos who only takes customers who marry her first, is Tysha. She has a 14 years old daughter named Lanna (Lanna is a Lannister/Westerlander name, feminine equivalent of Lann, founder of House Lannister) with long, fine golden hair, who could be Tyrion's daughter. So perhaps, there is some hope of a happy ending for Tyrion on that front.