Not really a plot-hole. You are forgetting the strategic, grooming and material advantages that Stannis and his Knights enjoyed as compared to ill-equipped, ill-disciplined rabble brought together by Mance Rayder.
King beyond the Wall did not bring 1,000 warriors with him. He brought 100,000. Of course it wasn't possible for the show-runners to show 100,000 warriors and it would have cost a lot in CGI if they were to digitally create the effect. HBO thinks a lot before showing dragons and tries to limit their appearance as much as possible due to the costs incurred in creating CGI of one to three dragons. Now imagine the cost when it comes to 100,000 warriors.

Nevertheless, Following is the force breakdown for Battle of Castle Black:
Wildlings:
- 100,000 warriors burdened with women (Except spearwives), Children and their supplies. Mostly infantry except few horsemen. They were ill-equipped with weapons made of bones , bronze and stones. They had no armor or experience in facing a charge by Knights.
- Giants and Mammoths
Southern Force:
- ~100 Defenders of Night's Watch safely behind the icy fortification called the Wall. They had the advantage of High Ground as well. Not to mention, they were trained, drilled and disciplined unlike their foe. They had steel weapons and possibly armor.
- ~2800 Baratheon Cavalry and Knights completely armored and mounted1. Trained and experienced in warfare. They had steel weapons and armor.
The Battle
Mance had devised a clever strategy of attacking Castle Black from behind and opening the gates for his Soldiers without any fight with defenders on the Wall.
The plan was especially clever because it was going to be a surprise attack. However, due to courage of a Night's Watch man, Jon Snow, the advantage of surprise was lost and Night's Watch became aware of the Wildling plans. They successfully defended their castle from a rear attack by Wildling force under Magnar of Thenn, Styr.
As it happens however, Mance's numerical superiority soon started to take its toll on the defenders who were over-stretched and exhausted due to lack of replacements or reinforcements.
That's when Stannis Baratheon came in.

Wildlings had made a critical error. They did not expect any help from the Kings of Westeros and therefore were expecting attacks only from the Wall. This lead to them leaving their Eastern flank unguarded.
Stannis was riding from Eastwatch-by-the-sea. He struck a pincer attack on Wildlings, many of whom were still sleeping or just getting up. He successfully broke through Wildling lines (Or whatever passed for them). Wildlings were completely surprised and they had no experience in how to stop a charge of thousands of mounted Knights, equipped with steel swords and armor. Baratheon men cut through them like knife goes through cheese.
Wildlings should have kept calm and formed a shield line with spears in front to stop the Knights but they did not know that because they had no military formal training or strategy. They only fought with brute-force and that was their sole plan.
Stannis on the other hand, doubled the surprise and panic when his second column hit the Wildlings from rear through the forest and Wildlings started running for their lives.
The movement of Baratheon Army shown below, in a pincer attack, enveloping the Wildlings:

Once an army starts running, their numbers do not matter. They are only meat for their pursuers.
Books' Perspective:
You haven't asked for Books' perspective but it will help you understand the battle better as compared to show.
This is how Jon described the battle in ASOS:
The free folk still had the numbers, but the attackers had steel armor and heavy horses. In the thickest part of the fray, Jon saw
Mance standing tall in his stirrups. His red-and-black cloak and
raven-winged helm made him easy to pick out. He had his sword raised
and men were rallying to him when a wedge of knights smashed into them
with lance and sword and longaxe. Mance’s mare went up on her hind
legs, kicking, and a spear took her through the breast. Then the
steel tide washed over him.
It’s done, Jon thought, they’re breaking. The wildlings were running, throwing down their weapons, Hornfoot men and cave dwellers
and Thens in bronze scales, they were running. Mance was gone,
someone was waving Harma’s head on a pole, Tormund’s lines had broken.
This is what Ser Thoren Smallwood had to say about Wildling numbers in ACOK:
“The numbers would be greatly against us,” Ser Ottyn had objected.
“Craster said he was gathering a great host. Many thousands. Without
Qhorin, we are only two hundred.”
“Send two hundred wolves against ten thousand sheep, ser, and see what happens,” said Smallwood confidently.
Conclusion:
Stannis Baratheon should have prevailed that day because:
- His forces were better equipped.
- His forces were better trained.
- He employed a successful envelopment strategy.
- His forces were all mounted and armored.
- His forces enjoyed the advantage of surprise.
- His forces enjoyed speed and operational maneuverability.
- His foes failed to break the charge.
- His foes did not have effective weapons to penetrate Baratheon armor.
- His foes formed no ditches and stakes to guard their encampment against any possible attack.
- Key Wildling chieftains who were important lieutenants of King Beyond the Wall were not present in the action. Styr was killed, Tormund was taken captive, Orell was also killed in his human form prior to the Battle of Castle Black.
Even if Wildlings had been able to repulse the Baratheon charge a few times (Despite their strategic and material incapacity to do so). Baratheons did not have to break themselves against the same force again and again. They were all horsemen. They could strike, fall back, go around and hit the Wildlings somewhere else, wrecking Havoc in Wildling lines everywhere on the front. Wildlings would have to deploy and redeploy their best men all over the front, which will take a lot of time given that they were almost exclusively infantry. By the time they deployed on new front, Baratheons could simply break away and hit somewhere else, giving the freefolk a merry chase. Result would have only been victory for Baratheons and defeat for the Free Folk.
If the Battle had been between any regular Army from South and 2800 Baratheons, the result would have been very different. The Southern Army would have broken down cavalry charge of Baratheons with spears and then slaughtered them due to lack of supporting infantry in Baratheon ranks. But the battle was not between Southerns. It was between the Southerns and the Notherners (Wildlings) who were utterly unprepared to stop the heavy cavalry and panicked at the first sight of Baratheon standards.
Quoting Ned Stark:
In Battles, Discipline beats numbers nine times out of ten.
1. Based on a count of cavalry squares visible on-screen, divided evenly between the two pincers. Stannis said four episodes before in "The Laws of Gods and Men" that he has about 4,000 men left overall; others must have been left at Dragonstone, bringing the baggage train overland from Eastwatch, didn't have horses, etc.