I don't have any canon answer, but from what I can tell, is that people in the South have problems of their own.
You must remember that the Targaryen regime wasn't really peaceful. Off the top of my head I remember that there was a civil war (The Dance of Dragons) as well as the Mad King. These two reasons, really had a bad influence on people. The civil war lasted many years and costed a lot of money and lives. During the reign of the Mad King, there was a revolution and then some peaceful years with King Robert. After that, more war.
People of the south needed to rebuilt and repopulate. They couldn't be bothered with something that was supposedly a threat many many miles away. They needed money for their own needs. After many wars,rebellions and fool kings, the people don't have as much money as they used to.
The wildlings never actually gathered together to form a good attack, like they did on the final episodes of season 4 under Mance Rayder's leadership. So, we can assume that the people of the south thought that even if they managed to get through the wall, their army could destroy them.
While Yoren was in King's Landing, he asked for help, (which if I'm not mistaken included money), but all his Majesty could spare were half a dozen criminals.
TL;DR: The wildlings always incapable of pulling off some strategical plan; the Black Brothers could most of the times take care of these attacks, so people didn't feel the need to fund them more than they already did.
As far as the Others go, people (even the Maesters) only believe that they are tales to scare children. No one alive had actually seen one, until the beginning of the show/book series.