"The Skills of Xanadu", a novelette by Theodore Sturgeon, which you can read at the Internet Archive; also the answer to this old question, among others.
The anthology you read it in in the 60s was probably 13 Great Stories of Science Fiction, edited by Groff Conklin.
I don't find the line "Don't confuse 'doesn't' with 'can't'" or anything like it in the story. Otherwise, it matches your description.
The planet Xanadu is visited by one man named Bril, an agent of a militant planet called Kit Carson:
Bril emerged near the pink star, disliking its light, and found the fourth planet. It hung waiting for him like an exotic fruit. (And was it ripe, and could he ripen it? And what if it were poison?) He left his machine in orbit and descended in a bubble. A young savage watched him come and waited by a waterfall.
The natives do not go naked, though Bril considers their attire scandalously revealing:
"I am Bril of Kit Carson, second planet of the Sumner System, and a member of the Sole Authority," said the newcomer, adding, "and I come in peace."
He waited then, to see if the savage would discard any weapons he might have, according to historic protocol. Wonyne did not; he apparently had none. He wore only a cobwebby tunic and a broad belt made of flat, black, brilliantly polished stones and could hardly have concealed so much as a dart. Bril waited yet another moment, watching the untroubled face of the savage, to see if Wonyne suspected anything of the arsenal hidden in the sleek black uniform, the gleaming jackboots, the metal gauntlets.
The natives have integrated nature with their environment:
It seemed to have no margins. It was here high and there only a place between flower beds; yonder a room became a terrace, and elsewhere a lawn was a carpet because there was a roof over it. The house was divided into areas rather than rooms, by open grilles and by arrangements of color. Nowhere was there a wall. There was nothing to hide behind and nothing that could be locked. All the land, all the sky, looked into and through the house, and the house was one great window on the world.
[. . . .]
The room was wide, wider at one end than the other, though it was hard to determine by how much. The floor was uneven, graded upward toward one corner, where it was a mossy bank. Scattered here and there were what the eye said were white and striated gray boulders; the hand would say they were flesh. Except for a few shelf- and tablelike niches on these and in the bank, they were the only furniture.
The natives get their powers from the belt with black stones, as Bril learns when he puts one on:
"One of our few superstitions," said Tanyne. "It's the formula for the belts—even a primitive chemistry could make them. We would like to see them copied, duplicated all over the Universe. They are what we are. Wear one, Bril. You would be one of us, then."
[. . . .]
The first thing he was aware of was the warmth. Nothing but the belt touched him anywhere and yet there was a warmth on him, soft, safe, like a bird's breast on eggs. A split second later, he gasped.
How could a mind fill so and not feel pressure? How could so much understanding flood into a brain and not break it?
[. . . .]
He knew without question that he had the skills of this people, and that he could call on any of those skills just by concentrating on a task until it came to him how the right way (for him) would feel. He knew without surprise that these resources transcended even death; for a man could have a skill and then it was everyman's, and if the man should die, his skill still lived in everyman.
Xanadu defeats Kit Carson by giving them the belts which make them smarter:
And then, as the designers in Xanadu had planned, all the other segments of the black belts joined the first meager two in full operation.
A billion and a half human souls, who had been given the techniques of music and the graphic arts, and the theory of technology, now had the others: philosophy and logic and love; sympathy, empathy, forbearance, unity in the idea of their species rather than in their obedience; membership in harmony with all life everywhere.
A people with such feelings and their derived skills cannot be slaves. As the light burst upon them, there was only one concentration possible to each of them—to be free, and the accomplished feeling of being free. As each found it, he was an expert in freedom, and expert succeeded expert, transcended expert, until (in a moment) a billion and a half human souls had no greater skill than the talent of freedom.
So Kit Carson, as a culture, ceased to exist, and something new started there and spread through the stars nearby.