Before Angmar was formed, it seems as though Arnor should have prospered: No external threats, a prosperous ally to the South, Elvish advisors to the East and West... why was there a depopulation? Specifically, of Annuminas? This, while Gondor fared relatively well for quite some time.
At the council of Elrond, Elrond himself gives a pretty good overview of the collapse of the city of Annúminas. The city's rulers were ambushed and killed. On top of that, a large proportion of the the city's men had been away at war for nearly a decade. When combined with the loss of their ruling council and a lowering population this made them easy prey for bandits and raiders.
They abandoned the (now indefensible) city and moved their capital before ultimately disbanding as a people, spreading out to other cities and states:
"In the North after the war and the slaughter of the Gladden Fields the Men of Westernesse were diminished, and their city of Annúminas beside Lake Evendim fell into ruin; and the heirs of Valandil removed and dwelt at Fornost on the high North Downs, and that now too is desolate. Men call it Deadmen’s Dike, and they fear to tread there. For the folk of Arnor dwindled, and their foes devoured them, and their lordship passed, leaving only green mounds in the grassy hills." - LOTR : Fellowship of the Ring.
The Peoples of Middle-Earth (HoME Vol 12) gives us a bit more back-story on their decline as a people. The kingdom was already in decline, it was split into three parts (the King had three sons, each powerful enough to command their own factions) and the kingdoms warred among themselves until they ground each other down to nothingness:
After Earendur the Northern Kingdom of Amor was broken up, the sons of the king established smaller independent kingdoms. The direct line of the eldest son ruled the realm of Arthedain in the north-west; their city was Fornost. Annuminas became deserted owing to the dwindling of the people. The chief of the lesser realms were Cardolan, east of the Baranduin; and Rhudaur north of the Bruinen. Arthedain still claimed the overlordship, but this was disputed. Cardolan south of the Great Road and east of the Baranduin; and Rhudaur north of the Great Road between the Weather Hills and the Bruinen. There was often strife between the kingdoms; the chief matter of debate was the possession of the Weather Hills and the land westward thence towards Bree.
It's established in Appendix A to Return of the King that Arnor had split early (before the coming of the Witch-king and the rise of Angmar) into three separate petty kingdoms, who then proceeded to squabble amongst each other:
After Eärendur, owing to dissensions among his sons their realm was divided into three: Arthedain, Rhudaur, and Cardolan ..... In Arthedain the line of Isildur was maintained and endured, but the line soon perished in Cardolan and Rhudaur. There was often strife between the kingdoms, which hastened the waning of the Dunedain.
Appendix B dates this to the year 861 of the Third Age:
Death of Eärendur, and division of Arnor.
A fuller account of this with specific reference to Annuminas is given in the material in History of Middle-earth 12 concerning the Kings of Arnor, where it is said:
After Earendur the Northern Kingdom of Arnor was broken up. The sons of the king established smaller independent kingdoms. The direct line of the eldest son ruled the realm of Arthedain in the north-west; their city was Fornost. Annuminas became deserted owing to the dwindling of the people.
This is consistent with the original Tale of Years material for the Third Age, the full form of which is also given in History of Middle-earth 12 and which says (again for the year 861):
Death of Earendur last and tenth king of Arnor. The North-kingdom becomes divided among Earendur's sons. The direct line of the eldest son, Amlaith of Fornost, rules the realm of Arthedain. Annuminas is deserted.
So it's established that the desertion of Annuminas didn't happen until the year 861, and that the direct cause was the splitting of the realm.
Aragorn, of course, re-established Annuminas during his reign, as we also learn from Appendix A:
Our King, we call him; and when he comes north to his house in Annuminas restored and stays for a while by Lake Evendim, then everyone in the Shire is glad.