As noted in the comments above, The Unfinished Tales has a section devoted to a thinly-veiled parable of Númenóreans imperialism during the years when their pride began to increase. At first they sailed to the defense of the Elves in the first War of the Ring (the Sack of Eregion), saving Eriador and the coast-lands from Sauron's conquest.
At this time, most of the Men in Middle-earth (including the distant relatives of the Edain) were still wild; this relationship is gone into more detail in the section of Unfinished Tales describing the Drúedain or Wood-Woses, the only indigenous (and non-white) tribe of Men who Tolkien really described. (When asked about this Tolkien basically said he preferred to write what he knows, and not about mythology of other cultures; he had strong emotional ties to Northern European myth.)
Despite this, the UT stories which were written late in Tolkien's life indicate that he had a lot of sympathy for the "wild" or "lesser" Men in Middle-earth, (lesser only in lifespan and their lack of ties to Elves) and the history of the "fall" of Númenor (and Gondor) is basically tied to their obsession with purity and increasing subjugation of "lesser" Men. This only causes their lifespan to decline further of course, since they didn't recognize mortality as a Gift and fought it.
Sometime in the early 2000s S.A. their settlements on Middle-earth went from centers of trade and cultural exchange (teaching wild Edain farming, and such) to resource extraction and colonization. This culminated in the building of great cities for purposes of imperial tribute; Only Pelargir (the original historic capital of what became Gondor) was settled primarily by Elf-Friends. (Elendil's line, which split off from the main line of kings after they rejected use of the Elvish language and took Adûnaic as the sole form of speech.)
It's also worth noting that the greatest of the Edain in the First Age were basically descended from wild men themselves, who had been uplifted by the Elves after choosing to ally with them in a suicidal battle with Morgoth. Morgoth had been worshiped throughout much of Middle-earth in the aftermath of the original Fall of Men. (Not depicted as Tolkien considered it analogous to the Biblical story and he didn't like explicit religious elements.)
Per another answer on this site, Morgoth was so powerful at the time of the War of the Jewels that it took the combined might of the Noldor to weaken him to a point where the Valar might hope to recapture him without devastating much of the fallen World in the process.
So the creation of Númenor was a reward for really any Men who dared to stand up to Morgoth, and the chapter on the Woses indicates it was open to all who did so (including some of the Woses). Their long lifespan was merely a restoration of the Gift of Men as originally intended before the Fall of men when they first awoke in the East (as described in the Athrabeth).
So when the Black Númenóreans returned to Middle-earth, they basically survived because they were expat colonial dictators at the time of the fall of Númenor, ruling their own kingdoms, possibly with their own rings (among the Nine); and given their worship of Morgoth, that's why they had a lot to work with.
Tolkien drops hints here and there that there was resistance to Sauron and Morgoth by Men in other parts of Middle-earth, but that it just wasn't successful. The Enemy held more sway in those areas because they were farther from the Noldor and their allies, and his tales only concerned the northwest of Middle-earth as it was impacted by the Elves.
That's why he didn't feel like continuing the story after the Dominion of Men when "mundane" history began. "Magic" was tied to the presence of Elves from the West; and the long life of the Men of the West was destined to fade as they merged with "lesser" (non-elvish associated) Men. The story of the fall of Númenor indicates that Tolkien didn't necessarily see this as a bad thing, since the Númenóreans who attempted to reverse this and maintain "racial purity" were almost universally Bad Guys (tm).
The Dúnedain who are most loyal to the ideals of their ancestors and have the most ties to Elves tend to be described as "in whom the blood of Númenor was strong" so it's a purity-of-heart thing. Conversely, the Rohirrim are closely related to the house of Hador but are depicted as noble savages compared to the Men of Gondor, akin to the ancient Anglo-Saxons; and the ancient tombs of the Edain on the Barrow-downs are venerated by the Dúnedain but also occupied by (Morgoth worshipping!) evil spirits, which suggests that the relatives of Men who didn't make it to Beleriand were not universally free from the Dark Lord's domination.