In the first part, we are told that Denethor had been a good and wise Steward and had "held the forces of Mordor at bay" for long. When Pippin and Gandalf visit Minas Tirith, he seems quite different from this description. He is very stubborn and irrational, as well as rude to even his own son. It appears he has lost his mind. Was Denethor always like this, or is there any particular in-text reason for the change in his behaviour?
Yes. The main reason (which the films only hint at), is that he has a Palantír (one of the seven seeing-stones, another one of which Saruman has). He knows that Sauron has one too, but he is overconfident and thinks he is powerful enough to control it.
In reality Sauron has been using it to show him only bad news (massing armies everywhere, lost battles, the ships of the Corsair sailing up the Anduin (Denethor doesn't know that they are actually under the control of Aragorn)). Sauron can't make a Palantír lie, but he can make it show you the part of reality that he wants you to see.
This drives Denethor to despair, thinking that all is already lost. Combined with the loss of his son Boromir, years of frustration over being just a steward instead of a king (aggravated by the fact that he knows the heir to the throne is coming), and finally the apparent death of Faramir, it pushes him over the edge.
He always loved Boromir and almost despised Faramir. He knows that his favourite son died on his way back from Rivendell. And now two persons with whom Boromir was traveling show up at "his" place. I think he just went mad with grief over Boromirs death.
Even if the official title was "Steward", Denethor and his family had been the only rulers of Gondor for almost one thousand years, since the death of King Eärnur.
In the book, when he meets Pippin and Gandalf, he is already aware that:
Boromir is dead
Together with Gandalf (almost by his hand) is Aragorn, who is going to claim the throne.
Has been ruling all his life
Is the last member of a long dynasty of (for anything except the name) kings
Was probably grooming Boromir as his heir.
And suddenly finds that:
he is going to lose his power
he is the last one of his dynasty,
his son is dead; all he knows about it is that the new King was supposed to be near him,
the throne is to be claimed by someone who, at most, has seen Gondor in the distance (and to top that, comes alongside the quite Machiavellian Gandalf)
there is none of the help his son was sent to ask for.
Additionally, Sauron pressure is intense and he does not know if Aragorn will be a help against it or will try to claim his title immediately (leading to internal dissent).
It is quite understandable that he was not very happy with anything related to the Fellowship. Sincerely, I would have thrown Pippin from the top of the walls.
Denethor is not so cool with Gandalf as his son Faramir, and Aragorn are for instance.
there was little love between Denethor and Gandalf; and after the days of Ecthelion there was less welcome for the Grey Pilgrim in Minas Tirith. [Appendix A: The Stewards]
Ecthelion was Denethor's father and Aragorn counseled him to trust in Gandalf, but Denethor
suspected that he and Mithrandir designed to supplant him. [The Stewards]
Gandalf warns Pippin to be on his toes in the presence of Denethor. So his mood and manner are not lost on Gandalf who has known him for over 80 years.
Théoden is a kindly old man. Denethor is of another sort, proud and subtle, a man of far greater lineage and power, though he is not called a king. [Return of the King; Minas Tirith]
I would not say that Denethor was irrational either. He was of sound mind even though he had been aged before his time. When he and Gandalf confront one another he tells him
to him there is no purpose higher in the world as it now stands than the good of Gondor; and the rule of Gondor, my lord, is mine and no other mans's, unless the king should come again. [Minas Tirith]
It has probably been ingrained in each Steward about the rule in Gondor that the King can only be someone who was descended directly in the male line from Anárion. Denethor himself held fast to this and he more likely than not knew who Aragorn was, perhaps not his true name, but he knew of his lineage and did not trust him. He was his "rival" in their early days in Gondor and also perhaps loved the power he had. Before he does kill himself he mentions
I am a Steward of the House of Anárion. I will not step down to be the dotard chamberlain of an upstart. Even were his claim proved to me, still he comes of the line of Isildur. I will not bow to such a one, last of a ragged house long bereft of lordship and dignity. [The Pyre of Denethor]
It should also be noted Gandalf came to Gondor thinking perhaps that Denethor had been turned. But this was not so. Sauron could not dominate him through the Palantir as he had done with Saruman. What did break Denethor in the end was the thought that his line was wiped out
my line too is ending, even the House of the Stewards has failed. Mean folk shall rule the last remnants of the Kings of Men. [The Siege of Gondor]