The North doesn’t have much—the land is difficult to farm, winters are especially brutal, there are some pockets of resources but nothing astounding (like Casterly Rock’s gold mines). The southern kingdoms have wealth, prestige, comfort, and power. The North has got... the Wall.
It’s unsurprising to me, then, that the North takes it seriously—it’s kind of like taking themselves seriously. If the Wall weren’t important, why on earth are they living there? Why did their ancestors live there? “Because they couldn’t get better land further south,” is an unpleasant, self-effacing answer—not one many people want to be the answer.
Likewise, the people of the south don’t want to feel like they owe the North anything. They don’t want to believe that the North is “sacrificing” or that their hardship is “for the greater good.” So they dismiss the Wall as unimportant. They do want to believe that they live in better lands because they and/or their ancestors were “better” in one way or another, and so got better land. That’s also a matter of pride.
And of course, in reality, both are right—the Wall is important, but getting stuck with guard duty also is rather getting the short end of the stick. It’s all a matter of perspective: are you honored to take on this duty and hardship, as the Starks are and have been? Or are you bitter about it, feeling slighted by it? They aren’t in the North, but Walder Frey and Balon Greyjoy might fall more into that latter category.
The North seems to follow the Starks’ example and take pride in it. Considering how difficult their lives are, those who don’t probably left—or lacked the willpower to endure and died out. They also likely saw social sanction if they questioned the importance of the Wall in front of their proud neighbors, which both makes their lives more unpleasant, and much more difficult if they lack friends to provide aid.
Any other consideration pales in front of this, I think. The wildling raids aren’t terribly severe and despite the rather poor state of the Night Watch at the time of A Song of Ice and Fire, they were quite capable of dealing with them
until Mance Rayder, but then, before the events of the books forced issues, not even the wildlings themselves took seriously the idea that someone could actually unite them against the crows
(minor spoiler from A Storm of Swords about what the wildlings have been up to)
The North would have had to devote far more resources to defense if the Wall didn’t exist, of course, but it did exist, and its existence alone would have been sufficient to allow a minimal number of men to keep raids down to a minimum—the North probably wouldn’t have ignored the Wall, but if wildling raids were the only reason to worry about it, they probably would have treated service there much more contemptuously, as the southern kingdoms did.
The Others were long forgotten and even the North didn’t take them seriously.