this is an old discussion but I'm reading Fiasco (in English, I sadly don't speak Polish) for the third time now and I'd like to leave my ideas:
The nature of the Quintans is deliberately left mysterious and ambiguous, I don't think Lem wanted the reader to be able to come to a full picture of what they really were because the point is that we can't ever really understand the truly alien. However, it's possible to get some idea of what Lem had in mind for them by looking at some of his other books and certain clues in Fiasco.
At one point the Hermes scans the planet and finds large areas of bone-like structures on the surface. This brings to mind his earlier novel Eden, where the "Doubler" civilization kept large buildings full of unformed skeletons create as a result of a massive failed eugenics project. I think this is one good possibility for why the Quintans are immobile mounds: they genetically engineered themselves into that form for whatever reason. It's possible that they were more mobile in the past. In Eden, the eugenics project is paired with "procrustics", an advanced attempt at information control where the organism is adapted to a standard and compelled to self-regulating oppression.
Meanwhile, the Quintan "warsphere" occurs in at least two other novels: peace on Earth and The Invincible. In both, the swarm is the result of "auto-evolution", a process Lem likens to cancer where a technological process that is initially controlled loses its teleological bearings and becomes a runaway process with its own internal logic, essentially a rebirth of natural processes within artificial ones. The cancer metaphor occurs in several places in Fiasco as well, and the prologue on Titan is full of imagery of "wild growths" of nature, the result of undirected, raw nature asserting itself.
Finally someone noted that nobody has been able to account for the ice ring, but something similar can, again, be found in another novel: the Star Diaries. On Dichotica, the originally human-like aliens have also fallen into an auto-evolutionary trap, whereby they have transformed their bodies into horrible forms as the result of a kind of informational discourse and an attempt to reach immortality. Ijon Tichy initially finds their planet because it is surrounded by a ring of garbage, the detritus of their auto-evolutionary process. This may be what the ice ring is: the Quintans changed their body shapes, and since they are anchored on land they need more land for increasing their population, so they threw their ocean into space as a kind of waste product.
Someone above posted that the novel seems to suggest that the mounds are in fact single organisms instead of colonies, but I'm not sure if the quoted passage really rules out the possibility of a colonial organism, but I think it's pretty strong evidence (along with the physical description of the mounds' interior, which is like "ropy dough").
All of this leads me to believe that the Quintans subjected themselves to a eugenics project, the result of some misguided attempt to "improve" themselves or to control information and dialectics, or because their technology required them to take such a shape. The "warsphere" is the technological product of the same autoevolutionary process. Ultimately this results in the disintegration and scattering of the individual intelligence, like on Eden (whose inhabitants are not exactly individuals but are described as "binary"), and on Dichotica (where individuals are physically shaped by the whims of a runaway process, in the same way fashion dictates what our clothes look like), and in The Invincible (where a swarmlike techno-biology has stripped away the natural biosphere completely).
I hope this makes sense, it's not intended as "proof" of what the Quintans are, rather I tried to synthesize elements that are made more clear in Lem's other novels into a theory of what happens in Fiasco.