Let me offer an somewhat undeveloped reason why short stories might be more important in science fiction than in other genres: the underlying landscape of science fiction isn't fixed.
To expand, note that if you want to write a western, you know what is available in the universe, the same with a hardboiled detective story or most other mysteries. Even if you throw in a twist in the form of a relatively unknown subculture, the rules haven't changed.
In science fiction and a few other genres (fantasy, horror) the rules are almost entirely in the authors hands, but there is a understanding between the writers and the readers of what kinds of rules to expect, and how to state what the rules are or to signal that we're playing outside the usual boundaries.
So ask, what if you pour you heart into a story with a new set of rules and the story never comes together, or it comes together but the readers hate it? If that story was a novel you've lost an enormous amount of effort and money. But, if it was a short story or novelette the investment is much smaller. And a similar calculation applies to readers picking up a work based on new(ish) ideas.
So, short fiction is a safe way for authors and readers to explore the available rules space.
But this means that most of the "important" ideas get told for the first time in short fiction. If it is to have a non-trivial shelf-life, it will have to be republished as part of a book. Thus, lots of anthologies.