Considering Arnor was made up of men descended from Númenoreans, one would think they would be a match for the forces of Angmar, or did Arnor not simply have enough numbers?
There are a number of reasons for this.
First of all, the kingdom of Arnor wasn't a single united kingdom, but split (it's important that this split happened in the time before the Witch-king):
After Elendil and Isildur there were eight High Kings of Arnor. After Eärendur, owing to dissensions among his sons their realm was divided into three: Arthedain, Rhudaur, and Cardolan.
Then, the three petty-kingdoms began to squabble between each other (it's also notable that the line of Isildur was only maintained in one of the petty-kingdoms):
In Arthedain the line of Isildur was maintained and endured, but the line soon perished in Cardolan and Rhudaur. There was often strife between the kingdoms, which hastened the waning of the Dunedain.
Then Rhudaur, as the weakest of the three, was easily overcome and dominated:
In the days of Argeleb son of Malvegil, since no descendants of Isildur remained in the other kingdoms, the kings of Arthedain again claimed the lordship of all Arnor. The claim was resisted by Rhudaur. There the Dunedain were few, and power had been seized by an evil lord of the Hill-men, who was in secret league with Angmar.
From there we get a bit of divide-and-conquer:
A great host came out of Angmar in 1409, and crossing the river entered Cardolan and surrounded Weathertop. The Dunedain were defeated and Arveleg was slain. The Tower of Amon Sul was burned and razed; but the palantír was saved and carried back in retreat to Fornost, Rhudaur was occupied by evil Men subject to Angmar, and the Dunedain that remained there were slain or fled west Cardolan was ravaged.
The Great Plague didn't help matters much either:
In the days of Argeleb II the plague came into Eriador from the Southeast, and most of the people of Cardolan perished, especially in Minhiriath. The Hobbits and all other peoples suffered greatly, but the plague lessened as it passed northwards, and the northern parts of Arthedain were little affected. It was at this time that an end came of the Dunedain of Cardolan, and evil spirits out of Angmar and Rhudaur entered into the deserted mounds and dwelt there.
By now Arthedain is left on its own:
In 1974 the power of Angmar arose again, and the Witch-king came down upon Arthedain before winter was ended. He captured Fornost, and drove most of the remaining Dunedain over the Lune; among them were the sons of the king.
So it's time to bring on the Easterlings:
But neither kingdom was able to send help to the other; for Angmar renewed its attack upon Arthedain at the same time as the Wainriders reappeared in great force.
And then Arthedain is destroyed:
Then Eärnil sent his son Eärnur north with a fleet, as swiftly as he could, and with as great strength as he could spare. Too late. Before Eärnur reached the havens of Lindon, the Witch-king had conquered Arthedain and Arvedui had perished.
All references: Lord of the Rings, Appendix A.
Finally, it's wrong to think that the population of Arnor was exclusively Dunedain: it wasn't. The Realms in Exile were actually mostly made up of indigenous people, who the Númenoreans just ruled over.
Darth Satan's answer is correct, but I would like to summarize and highlight a few things. Mainly, I disagree with the idea that Arnor was defeated easily.
Angmar was founded around TA 1300, with the intention of destroying the men in the North. The "evil men" allied with Angmar had a strong presence in Rhuduar, so when Angmar attacked Cardolan and Weathertop in TA 1409, they had the aid of those men and were able to wipe out the remaining Dúnedain in Rhuduar and effectively destroy the kingdom of Cardolan as well. So at this point, only Arthedain was left to effectively resist Angmar.
Despite standing on its own (Gondor was an ally, but could only provide limited aid due to attacks from the Easterling Wainriders), Arthedain held off the forces of Angmar for more than 500 years. It wasn't until TA 1974 that the capital of Fornost was captured and Arthedain was destroyed.
So, from its founding, it took almost 700 years for Angmar to achieve its final victory of Arnor. Also, Angmar itself was defeated and the Witch-King overthrown only a year later, in a counter-attack from Gondor, Lindon, and Rivendell. So, while Angmar was successful in its goals, it was hardly an easy victory.