I believe the vows of the Night's Watch go back through the ages, in the order of thousands of years, back to mythical heroes like Brandon the Builder and before him. GRRM has allowed his characters somewhat free interpretation of what these vows mean.
Sam (and Bran's group) was made aware of the old gate, under the Nightfort, which tied the vow to the gate's magical quality. I believe there are more such ancient magical qualities tied to the vow. There seems to be some purpose to Jon and Sam saying their vows before a heart tree, for example.
Many black brothers seem to believe that protecting the realm from wildlings is the core of the oath, whereas we have seen that Stannis and Jon have made the interpretation that their duty is to protect the realms of men, in which wildlings are considered on the side of the realm. I believe this comes close to the original intent of the vow. As Lord Commander Mormont said (paraphrasing):
"You don't build a wall hundreds of feet high to keep out raiders."
In the time when the oath was conceived, I believe the intent was to form an elite guard force, whose sole task was to guard against the ancient evil, the Long Night, The Great Other. As is mentioned in many of Jon's chapters, I cannot remember if it was Maester Aemon or Lord Commander Mormont, family ties are strong, and the ones who test a man's oath the hardest.
While other men might be part of the realm; Lords, servants, knights, fathers, etc, the Night's Watch stands outside it. They have severed all ties to it. Their sole focus is the danger north of the wall. It is in their name: The Night's Watch, they who watch against the Long Night.
Night gathers, and now my watch begins.
They are watchmen. Not primarily soldiers, or an army.
It shall not end until my death.
Making this pledge binds you forever, so that you shall never have any cause to think of your past or your future.
I shall take no wife, hold no lands, father no children. I shall wear no crowns and win no glory.
This is the meat of the pledge: To forever forsake your claim to your place in the realm.
I shall live and die at my post. I am the sword in the darkness. I am the watcher on the walls. I am the fire that burns against the cold,
the light that brings the dawn, the horn that wakes the sleepers, the
shield that guards the realms of men. I pledge my life and honor to
the Night's Watch, for this night and all nights to come.
This last part to me has always revolved around "the watcher on the walls", "I shall live and die at my post" and "the horn that wakes the sleepers," -- which also ties into the very dramatic 1-2-3 blasts of their horn signals: Once for rangers returning, twice for wildlings, three times for Others.
While other people, such as for example the Lord of Winterfell and his men, may take up arms against the enemies of the realm, they are not pledged to guard against the Long Night, their attention may be divided. They might enjoy coming home a cold evening, sitting in front of their fire with their children and turning a blind eye to the dark and cold night outside.
But someone has to be the one to keep watch, to wake the sleepers, to bear witness to strange events, to listen to the forest and sense the creeping cold. Someone who spends his life in the cold and empty forests and learns how to read its signs.
Much like in the prologue to AGOT, Gared and Will could sense something was wrong, but Ser Waymar Royce could not.
I guess in short it boils down to no division of loyalty and no distractions. Much like any knight can protect the king in his own right, and has his own place in the protection around the king, a knight of the Kingsguard has the duty of being the first and innermost line of defense. His watchfulness can never waver for a second, as Ser Barristan notes in ADWD:
But the battle was never truly done for a knight of the Kingsguard. Threats came from everywhere and nowhere, at any time of day or night. [....] For every hour of fighting, a Kingsguard knight spent ten thousand hours watching, waiting, standing silent in the shadows. King Hizdahr's pit fighters were already growing bored and restive with their new duties, and bored men were lax, slow to react.
I think Lord Commander Mormont demonstrated it the best. On their ranging, he comes to the conclusion that something very bad is happening, that the wall and the Night's Watch's purpose is not what they thought. He was able to make that deduction because his only focus in life is being the watcher on the walls. That guard who has spent thousands of hours watching and recognizes the danger in time.