My obsession with science-fiction first came to me through short stories, read in anthologies and single-writer collections.
Dangerous Visions and The Science-Fiction Hall of Fame are more or less required reading, and it seems as if the SF shelves always have more than their share of anthologies.
Why are anthologies such an important part of the history of science-fiction? Is this specific to the SF and fantasy genres? Is there some reason that publishers keep putting these out?
I think that the place of the anthology in the history of science fiction is a factor of the place of magazines in that history. Early science fiction was developed and nurtured in the pulp magazines like Amazing Stories. (Which was edited by Hugo Gernsback, the man after whom the Hugo Award was named.)
Writers for Amazing Stories included E.E. "Doc" Smith, John Wyndham, Fritz Leiber, and Clifford D. Simak.
Other major pulp magazines included Astounding Stories, which later was renamed to Analog Science Fiction and Fact; Wonder Stories; and Weird Tales. Particularly in the 1920s-40s, they were pretty much the only ones paying for science fiction writing.
Starting in 1937 editor for Astounding Stories was John W. Campbell. His name was used for the Campbell Award for best new science fiction writer. Asimov called Campbell "the most powerful force in science fiction ever." He published the first stories from Lester Del Rey, A.E. van Vogt, Theodore Sturgeon, and Robert Heinlein (who probably doesn't need the link, but I like to be complete). He was also the one who insisted on the ending for "The Cold Equations."
The anthologies you're referring to are, for the most part, re-publishing the work from these magazines. It's really the pulp magazines (and their editors) that influenced the creation of the field.
Dangerous Visions is a different case -- that was all original stories, and Harlan Ellison allowed many things that were forbidden in the magazines, such as sex and swearing. I think he was trying to see what might have come about in the magazines if they didn't have those restrictions. But an anthology like that could only be printed when there was already an audience for SF, which is why it came out in 1967, decades after the start of the pulps.
To answer the easy bit first, publishers keep putting out anthologies because they think they can make money by doing so. Although, in fact, there are precious few original anthologies published these days.
Your real question is, why are there SF short stories at all. The short story has practically died out, except in SF. I think this is partly because SF is about ideas, and a short story is a good length for exploring one specific idea, and partly because there are so many short fiction awards in the SF field. And once those short stories exist, people are going to want to reprint them.
Let me offer an somewhat undeveloped reason why short stories might be more important in science fiction than in other genres: the underlying landscape of science fiction isn't fixed.
To expand, note that if you want to write a western, you know what is available in the universe, the same with a hardboiled detective story or most other mysteries. Even if you throw in a twist in the form of a relatively unknown subculture, the rules haven't changed.
In science fiction and a few other genres (fantasy, horror) the rules are almost entirely in the authors hands, but there is a understanding between the writers and the readers of what kinds of rules to expect, and how to state what the rules are or to signal that we're playing outside the usual boundaries.
So ask, what if you pour you heart into a story with a new set of rules and the story never comes together, or it comes together but the readers hate it? If that story was a novel you've lost an enormous amount of effort and money. But, if it was a short story or novelette the investment is much smaller. And a similar calculation applies to readers picking up a work based on new(ish) ideas.
So, short fiction is a safe way for authors and readers to explore the available rules space.
But this means that most of the "important" ideas get told for the first time in short fiction. If it is to have a non-trivial shelf-life, it will have to be republished as part of a book. Thus, lots of anthologies.
I think anthologies are such an important part of science fiction in part because so many authors get started writing short stories. These stories are then published in magazines. As the author becomes more popular, it becomes difficult to find these stories and so anthologies are published. Isaac Asimov stated in one of his anthologies (I believe it was Gold) that the reason he had so many was because his agent pointed out that every few years you have a new generation of people who never had the chance to read the original stories, and so a new anthology is published. Other anthologies are written to keep the word going about good science fiction stories.
tl;dr: Science fiction anthologies are put together to keep the word going about good stories whether they be author specific or theme specific.
This question was answered by Fletcher Pratt in the opening paragraphs (quoted below) of his introduction to Martin Greenberg's 1951 anthology Journey to Infinity:
Science fiction is unique among the modern groupings of literature in that the anthology is perhaps its most typical form. Not that there are no book-length novels in the field. There are some extremely good ones; but the method and material of science fiction lend themselves particularly well to the short story. The writer of science fiction is and must be concerned with the reactions of human beings in environments which, either by time or circumstance, are strikingly different from the world in which we spend our daily lives. If that writer elaborates the picture of his imagined world in every last logical detail, he risks losing track of the individual people he is writing about. That is, he turns out a treatise instead of a story; and in fact much book-length science fiction is more science than fiction, Bellamy's Looking Backward being a famous example. The writer who is really telling a story can normally afford only a glimpse of the different world in which his tale is laid, enough to indicate its main lines and why it is provided with pleasures, duties and perils unlike those that normally surround us.
Moreover, science fiction appears largely in magazines for the first time, and the modern American magazine reader has established his perfectly reasonable repugnance to being bothered either with very long stories or losing the thread of a type of story that always requires rather close reading while waiting for the next issue to come out. The short story has thus come to dominate the field of science fiction, and it is not surprising that various people have found many short stories too good to be left gathering dust in piles of back number magazines.
