Night's Watch were by no means incompetent so the basic premise of the Question is flawed.
Jeor Mormont's decision to lead 300 men into the wild is not a strategic misstep. You are miscalculating the defending capabilities of the wildlings. Wildlings were no match for discipline and equipment of the men of the Night's Watch. Also if Wildlings climbed across from any place unseen (Which they would have done if Night's Watch did not send rangers beyond the wall), Castles would be completely vulnerable as they had no defenses in the South.
I'd like to split my answer into following parts.
Night's watch was constantly losing rangers on their reconaissance patrols. Which meant something was happening beyond the wall and someone did not want the Night's Watch to know what was going on. Jeor Mormont could either stay put and be surprised by whoever it was on their ground of choosing and at their time of choosing. That could have spelled the end of the Night's Watch. Or he could ride outside and find out for himself what was going on. The enemy who was picking out rangers one at a time would find a tougher foe in 300 trained men of the NW.
There were more dire things happening as well. Joer Mormont had survived an attack by wights. Night's watch had forgotten about that enemy and this event meant something bigger than wildlings was at play in wilderness beyond the wall. Night's watch was sworn to defend the realms of men and wake the sleepers in the South in time of danger. How could Night's watch do this if they didn't know themselves what was the danger?
Quoting the strange events reported to Lord Commander by then from AGOT:
The cold winds are rising, Snow. Beyond the Wall, the shadows
lengthen. Cotter Pyke writes of vast herds of elk, streaming south and
east toward the sea, and mammoths as well. He says one of his men
discovered huge, misshapen footprints not three leagues from
Eastwatch. Rangers from the Shadow Tower have found whole villages
abandoned, and at night Ser Denys says they see fires in the
mountains, huge blazes that burn from dusk till dawn. Qhorin Halfhand
took a captive in the depths of the Gorge, and the man swears that
Mance Rayder is massing all his people in some new, secret stronghold
he’s found, to what end the gods only know. Do you think your uncle
Benjen was the only ranger we’ve lost this past year?
So the ranging was necessary for following reasons:
- They needed to find out why wildlings were abandoning their villages.
- They needed to find the missing rangers or whatever happened to them.
- They needed to find out the dead men rising in the north.
- They needed to find out what was Mance Rayder up to.
Why did Lord Commander choose to ride out?
Jeor Mormont explained his decision to ride out in these words:
“I will not sit here meekly and wait for the snows and the ice winds.
We must know what is happening. This time the Night’s Watch will ride
in force, against the King-beyond-the-Wall, the Others, and anything
else that may be out there. I mean to command them myself.”
The first rule of battle is never to give your enemy what he wants. Jeor Mormont took the initiative and moved on the enemy before the enemy could move on them.
As Blackfish Bryden Tully explained to Catelyn Stark:
My first rule of war, Cat, is never to give the enemy his wish.
"Friends" beyond the wall
Night's Watch had friends beyond the wall like Craster. They had hoped to find some intel from him but since rangers were getting picked out by an unseen enemy, it was unlikely that any small party would be able to meet Craster and return to the wall. From Jon's POV in ACOK:
Thoren Smallwood swore that Craster was a friend to the Watch, despite
his unsavory reputation. “The man’s half-mad, I won’t deny it,” he’d
told the Old Bear, “but you’d be the same if you’d spent your life in
this cursed wood. Even so, he’s never turned a ranger away from his
fire, nor does he love Mance Rayder. He’ll give us good counsel.”
Night's Watch superiority
Night's Watch was superior to Wildlings in equipment, training and discipline. Wildlings lacked all of these. Quoting from Jon's POV in ACOK:
“The numbers would be greatly against us,” Ser Ottyn had objected.
“Craster said he was gathering a great host. Many thousands. Without
Qhorin, we are only two hundred.”
“Send two hundred wolves against ten thousand sheep, ser, and see what
happens,” said Smallwood confidently.
Wildlings fought like heroes. Brave yes, but without any strategy. They fought for glory and as many men as they could take down with them; ultimately, a good death. Night's Watch fought with strategy and discipline. They knew when to retreat, when to double back, when to attack.
Wildlings had poor weapons made of stones and bones. At max they had few bronze weapons and a few stolen steel swords. Night's watch was equipped with best quality steel weapons and armor. They were all mounted and thus had the advantage in mobility. Wildlings were further hampered by women and children. Night's Watch was not hampered by anything. They could cut through the wildlings like knife goes through cheese. They did not have to fight a pitched battle. They could hit and run. Then come back, hit again and run again. Wildlings would be harassed all the way to the wall and then there was wall itself.
Bone and Bronze and Copper are no match for castle forged steel. Stannis proved that with 1,500 men when he forced a general rout of the entire Wildling host. If 1,500 Baratheon men can prevail over 100,000 wildlings in pitched battle, 300 NW men can also cause considerable damage with hit and run tactics. From ASOS, Jon's POV:
The free folk still had the numbers, but the attackers had steel armor and heavy horses. In the thickest part of the fray, Jon saw
Mance standing tall in his stirrups. His red-and-black cloak and
raven-winged helm made him easy to pick out. He had his sword raised
and men were rallying to him when a wedge of knights smashed into them
with lance and sword and longaxe. Mance’s mare went up on her hind
legs, kicking, and a spear took her through the breast. Then the
steel tide washed over him.
It’s done, Jon thought, they’re breaking. The wildlings were
running, throwing down their weapons, Hornfoot men and cave dwellers
and Thens in bronze scales, they were running. Mance was gone, someone
was waving Harma’s head on a pole, Tormund’s lines had broken.
Night's Watch objectives
The objective was not to destroy the Wildling army. It was to buy time for the men at Wall to prepare defenses and notify the lords of the realm. They also needed to find exact info on what they were facing. As said by Qhorin Halfhand to Lord commander in ACOK:
“Belike we shall all die, then. Our dying will buy time for our
brothers on the Wall. Time to garrison the empty castles and freeze
shut the gates, time to summon lords and kings to their aid, time to
hone their axes and repair their catapults. Our lives will be coin
well spent.”
Wall's Defensive Value
You are thinking only about defensive value of the wall but you have forgotten that all but three castles on the wall were abandoned. There were not enough men to mount efficient patrols. If Night's Watch had chosen to stay blind in their castles, they would never have known about the rear attack on Castle Black by Thenns and Freefolk. They would never have been ready to defend the Castle Black. They would never have had the time to ask for aid from East Watch, Shadow Tower and Lords of the realm.
If the strike from South had been a success, Wall's height would have meant nothing. Wildlings would have slaughtered the garrison in their beds and opened the gates for Mance Rayder. Instead, thanks to the ranging, Night's Watch was ready for the southern attack and main invading force. So it was a great strategic move by Jeor Mormont which actually saved the Night's Watch from defeat.
Conclusion
In any case, Night's Watch did not have sufficient men to defend the wall in case of an invasion. It only made sense to ride out and find the danger to alert the realm.
Night's Watch enjoyed advantage in discipline and technological superiority. They were well-armed and armored unlike the wildlings so they were ten times as dangerous as a wildling. In a pitched battle of course this advantage will be neutralized by numerical superiority of the wildlings but that was not the strategy the black brothers chose. They chose to hit and run once the rangers returned. In the meanwhile they were waiting for the wildlings at fist of the first men. Which according to Mance:
“Aye, he was [Ready for me]. Had I been fool enough to storm this
hill, I might have lost five men for every crow I slew and still
counted myself lucky.”