I think I remember a line in A Clash of Kings or A Storm of Swords where some guy talked about why (at least, why he thought) the Night's Watch wore all black. I think it had to do with something about when they die? I can't quite remember, and it's bothering me. On the other hand, perhaps I fabricated this memory entirely or I'm mixing it up with something else.
I think this has to do with the original purpose of the Night's Watch, which is to protect the realm from the mysterious others from beyond the wall, who come out at night. That's why it is called the Night's Watch, and at night wearing black is the perfect camouflage.
Also, there is an interesting contrast (or balance?) between the brothers of the Night's Watch, who wear black, and the knights of the Kingsguard, who wear white. Both take pretty much the same vows to hold no lands and father no children, but their purpose is quite different.
“There are thousands,” someone called from behind Chett.
“We’ll die.” That was Maslyn’s voice, green with fear.
“Die,” screamed Mormont’s raven, flapping its black wings. “Die, die, die.”
“Many of us,” the Old Bear said. “Mayhaps even all of us. But as another Lord Commander said a thousand years ago, that is why they dress us in black. Remember your words, brothers. For we are the swords in the darkness, the watchers on the walls . . .”
Excerpt From: George R. R. Martin. “A Storm of Swords.” Bantam Dell, 2003-03-04. iBooks. This material may be protected by copyright.
Is that what you were thinking of?
The Night's Watch is on the order of 8000 years old and was formed at a time that was toward the end of a Long Night. It's probably safe to assume that they chose black back then to have perfect camouflage in the darkness / near darkness, and when the Long Night ended it had become a tradition that they chose to preserve -- Agreed to possible detriment, or perhaps to instill fear in their enemies by way of reputation.
As some other people brought up in response to suggestions that it was to camouflage them, my theory is the exact opposite, THAT IT IS MEANT TO STAND OUT AGAINST SNOW. In battles against white walkers, Night's Watch men are likely to be risen again if they die and the main force retreats, and it would be in the best interest of the brothers guarding the wall if fallen brothers stand out against the snow when they attack again as ghouls.
Black is the absence of Colors. Different houses in Westeros prefer different colors in their sigils, which become part of their identity. For example:
- Baratheon: Black on gold
- Stark : Grey on white
- Lannister: Gold on red
- Arryn: White on blue
Night Watch vowed not to take part in the wars of the Seven Kingdoms. They do not take any sides, hence the lack of colors.