In "The Lord of the Rings" and the "The Silmarillion" some Maiar (Sauron, several Balrogs, Saruman) are defeated. But did these Maiar actually die and if they did, do they go to Halls of Mandos or what happens to them? Since Morgoth was not actually killed by the Valar after the War of Wrath, I assume that it might be impossible to kill one of the Valar, but I'm not sure whether the same thing is true for the lesser Ainur.
That depends on what you mean by "die." I'll start with the assumption that Elves and Humans are considered to die. I would define that as their soul being severed from their physical body and that they can no longer independently assume a physical form. You will recall that Elves' souls go to the Halls of Mandos and Humans' out of the World to join Iluvatar.
That being established, let's review what happens to the Vala and Maia we read about. Their physical bodies can obviously be destroyed. The question then is "what happens to their soul?" I think it's clear that the Valar themselves can't be "killed" in the sense that when their physical form is destroyed they can just stay in their natural, non-corporeal form or appear in another form as they choose; this is why in order to defeat Morgoth once and for all the Valar had to "cast him into the void." It's not entirely clear what that means, but I don't think it's in any sense "death," just a sort of imprisonment or exile. When Sauron's first body was destroyed, he was able to create another body by himself, though he couldn't make it as fair as the first, which is why he took the rather more fitting form seen in the movies. As I recall, the Maiar, like the Valar, have "bodies" that are only a physical projection of their real selves. Gandalf and the other Istari are Maiar, but it seems rather less powerful than Sauron, and when Gandalf "died" in the fight with the balrog (another Maia), Eru had to intervene to reincarnate him (and the Balrog didn't come back at all), so I think there must be some sort of threshold of either power or will to continue to inhabit Middle-earth (probably a combination) that determines whether their spirit continues as if nothing happened.
So, from this definition and what we know about the Ainur whose bodies were destroyed, I think there is a threshold combining their power and will to live that determines whether they can create a new body for themselves.
As far as Sauron goes, especially his destruction in the war of the ring, I think in the end he put too much of his power into the One Ring, and when it was destroyed, that part of his power was destroyed along with it (similar to a horcrux in HP, though more power than soul), leaving him without enough power to continue to inhabit Middle-earth, create a new body, etc.
Some relevant passages
About the forging of The Ring:
but secretly Sauron made One Ring to rule all the others... And much of the strength and will of Sauron passed into that One Ring.
Sauron's first body was destroyed in the rending of Middle Earth, when the Valar removed Valinor from the world and destroyed Numenor:
The world was broken, and the land was swallowed up, and the seas rose over it, and Sauron himself went down into the abyss. But his spirit arose and fled back on a dark wind to Middle-earth, seeking a home.
And his creation of a new body:
There [Mordor] now he brooded in the dark, until he had wrought for himself a new shape; and it was terrible, for his fair semblance had departed for ever when he was cast into the abyss at the drowning of Numenor.
But Sauron also was thrown down, and with the hilt-shard of Narsil Isildur cut the Ruling Ring from the hand of Sauron and took it for his own. Then Sauron was for that time vanquished, and he forsook his body, and his spirit fled far away and hid in waste places; and he took no visible shape again for many long years.
When he returns again, as the sorcerer of Dol Guldur, it is only said that he "took shape."
When the ring was destroyed:
Then Sauron failed, and he was utterly vanquished and passed away like a shadow of malice
The 'death' of Saruman is the most descriptive:
To the dismay of those that stood by, about the body of Saruman a grey mist gathered, and rising slowly to a great height like smoke from a fire, as a pale shrouded figure it loomed over the Hill. For a moment it wavered, looking to the West; but out of the West came a cold wind, and it bent away. and with a sigh dissolved into nothing.
Frodo looked down at the body with pity and horror, for as he looked it seemed that long years of death were suddenly revealed in it, and it shrank, and the shriveled face became rags of skin upon a hideous skull.
This might be compared to the fall(s) of Sauron. When defeated by Isildur and the Armies of the West he is diminished but strong enough to rise again and the belief at the end of the War of The Ring is that Sauron will be too weak to come back.
One might suppose some religious overtones here (Tolkien was a devout Catholic) where the fallen Saruman is denied 'heaven' in the West beyond the sea.
Tolkien states in no uncertain terms, in Letter 156, that Gandalf did die:
Gandalf really 'died', and was changed...
The definition of 'death' here is, of course, all tied up in the definitions of 'mortal' and 'immortal' in Tolkien's work, and the Silmarillion probably summarises them best.
But this condition Iluvatar made, or it is the necessity of their love, that their power should thenceforward be contained and bounded in the World, to be within it for ever, until it is complete, so that they are its life and it is theirs. And therefore they are named the Valar, the Powers of the World. (Ainulindale)
To be immortal in Tolkien is not just to live forever, but to have your spirit bound within the world, until the end of the world. Although the above passage is written explicitly about the Valar, it also applies to the Maiar (who are also Ainur) and to Elves.
Therefore he willed that the hearts of Men should seek beyond the world and should find no rest therein; but they should have a virtue to shape their life, amid the powers and chances of the world, beyond the Music of the Ainur, which is as fate to all things else ..... It is one with this gift of freedom that the children of Men dwell only a short space in the world alive, and are not bound to it, and depart soon whither the Elves know not. (Of the Beginning of Days)
To be mortal in Tolkien is not just to die, but to also have your spirit free of the above binding to the world, as well as to have the ability to have control of your own destiny beyond that which is predestined for all immortal beings.
For the Elves die not till the world dies, unless they are slain or waste in grief ... and dying they are gathered to the halls of Mandos in Valinor, whence they may in time return. But the sons of Men die indeed, and leave the world; wherefore they are called the Guests, or the Strangers. (Of the Beginning of Days)
In the case of Gandalf, his spirit left the world, and in Tolkien that is the definition of "to die indeed":
Then darkness took me; and I strayed out of thought and time, and I wandered far on roads that I will not tell. (The White Rider)
It's well understood that Gandalf was sent back by Iluvatar, who had taken up the Valar's plan at the moment of it's failure, and that therefore Gandalf's spirit had most definitely left the world: this is "dying indeed".
Of course the Istari were a special case, since the bodies they inhabited were "real and not feigned" (Unfinished Tales, The Istari):
By 'incarnate' I mean they were embodied in physical bodies capable of pain, and weariness, and of afflicting the spirit with physical fear, and of being 'killed', though supported by the angelic spirit they might endure long, and only show slowly the wearing of care and labour. (Letter 156)
So to bring it all together, the answer is that no, in the normal course of events the Valar and Maiar cannot die as their spirits are bound to the world. Their bodies can be physically destroyed (in which case they just create a new one), or their spirits reduced to impotency, but being bound to the world does not permit the definition of "death" in Tolkien. The exception to this is the Istari (and possibly only Gandalf, as a result of direct intervention by Iluvatar).
If you adhere to the Canon of Ainulindalë, the answer is no - they cannot die, because of the conditions that Iluvatar states to the Ainur who wanted to descend into Ea:
"Thus it came to pass that of the Ainur some abode still with Ilúvatar beyond the confines of the World; but others, and among them many of the greatest and most fair, took the leave of Ilúvatar and descended into it. But this condition Ilúvatar made, or it is the necessity of their love, that their power should thenceforward be contained and bounded in the World, to be within it for ever, until it is complete, so that they are its life and it is theirs."
The Darkening Of Valinor in the Silmarillion also tells us that physical form is optional to Valar and Maia - though it seems to take some degree of effort. Melkor for example, at one point lost the capability to change form:
"For he [Melkor] was yet as one of the Valar, and could change his form, or walk unclad, as could his brethren; though that power he was soon to lose for ever."
It would therefore stand to reason this is what also happened to Sauron and Saruman.
The Valar can die, eventually or at least Melkor/Morgoth will die, the end of time was to be a battle between Fëanor, Tulkas, and the reincarnation of Túrin Turambar, and Túrin is to kill Melkor
So eventually Valar can die.
At the end of Quenta Silmarillion, Morgoth, is banished out of middle earth. If they could kill him they would have. Valar cannot die, but Maiar can. Though the latter would be hard, they don’t seem to get ill and they don’t suffer the negative effects of age, plus they are very powerful and some ( like Sauron ) take measures to stop themselves dying even then, like when Gandalf was saved by Illuvatar.