There's various factors:
1. They aren't actually expecting an attack on Asgard
The Vanaheim battle was presented as something of a conclusion to the Asgardian campaign to re-establish order in the 9 realms and after that there weren't any forces out there that they considered a credible threat. The Dark Elves were considered to be all but extinct and no-one had seen or heard from them in thousands of years. Even to the long-lived Asgardians this is a very long time, and since it was Odin's father that was fighting the war against them it's a pretty safe bet that none of the troops we see fighting in the palace have even seen a Dark Elf, let alone fought against one.
Add in that they were able to almost completely by-pass Asgard's early-warning system (i.e. Heimdal) and this means that they are right on top of the Asgardian defenders before they even realize what's happening. Everything about the Dark Elves' attack plan is also designed to maximize the confusion and keep the defenders off-balance - e.g. the infiltration of the Kursed elf into the prison. Having a force of strong and skilled enemies suddenly turn up in your palace and start ripping your home city and royal palace apart with powerful weapons that you've probably never seen before while your leaders/heroes are scattered around the complex and unable to provide clear direction is going to throw even well-trained troops off their game.
2. The Dark Elves aren't you're average mooks
It's highly likely that the "threats" they were dealing with while pacifying the 9 realms simply weren't up to the power or skill level of the Dark Elves and if you're used to coming up against amateurs you'll struggle when suddenly faced with a pro!
3. Asgard is no longer the warlike society it was at the time of the last war with the Dark Elves
We know from both Thor and Thor: Ragarok that before Odin mellowed somewhat Asgard was much more aggressive and warlike and was engaged in a fairly steady stream of wars and conquests. Then they had centuries of relative peace, it's not surprising that there would be a level of complacency allowed to develop. By comparison the Dark Elves have (from their perspective) are picking up where they left off.