probably from 60's .. planet is sentient, but since it's the only one it knows about, when a human expedition arrives it becomes paranoid, sending that fear out to the humans who then intensify it and send it back. Resolution comes when one of the crew runs off into the planet's forest, trips and hits his head so he goes unconscious. He is then able to mentally merge with the planet's "mind" and become a participant rather than an "other." That ends the feedback loop. He remains on the planet while the rest of the crew leave.
This is almost definitely LeGuin's "Vaster than Empires and More Slow" (1971).
A survey ship lands on a purely vegetated planet, World 4470. The crew of the survey ship feels fear broadcast at them from the forest even though there are no traces of intelligent life or even any animal life.
As she turned Osden's body over so they could lift him, his eyes opened. She was holding him, bending close to his face. His pale lips writhed. A deathly fear came into her. She screamed aloud two or three times and tried to run away, shambling and stumbling into the terrible dusk. Harfex caught her, and at his touch and the sound of his voice, her panic decreased. "What is it? What is it?" he was saying "I don't know," she sobbed. Her heartbeat still shook her, and she could not see clearly. "The fear—the... I panicked. When I saw his eyes."
"We're both nervous. I don't understand this—"
"I'm all right now, come on, we've got to get him under care."
Both working with senseless haste, they lugged Osden to the riverside and hauled him up on a rope under his armpits; he dangled like a sack, twisting a little, over the glutinous dark sea of leaves. They pulled him into the helijet and took off. Within a minute they were over open prairie. Tomiko locked onto the homing beam. She drew a deep breath, and her eyes met Harfex's. "I was so terrified I almost fainted. I have never done that"
"I was... unreasonably frightened also," said the Hainishman, and indeed he looked aged and shaken. "Not so badly as you. But as unreasonably."
The move the ship, but the fear remains (or follows them):
"How can the fear be here too?" she said, and her voice rang flat and false in the terrific silence. "It's not just the trees; the grasses..."
"But we're twelve thousand kilos from where we were this morning; we left it on the other side of the planet."
"It's all one," Osden said. "One big green thought. How long does it take a thought to get from one side of your brain to the other?
They decide that the problem is that the alien sentience has never before encountered an "other" and is responding in fear:
"Isolated," said Osden. "That's it! That's the fear. It isn't that we're motile, or destructive. It's just that we are. We are other. There has never been any other."
"You're right," Mannon said, almost whispering "It has no peers. No enemies. No relationship with anything but itself. One alone forever."
Osden, the team's empath wishes to communicate with the forest.
"If I gave in to it," Osden mused, "could I communicate?"
He seems to be successful, causing the forest's fear to end, and chooses to remain there where he won't be subject to the fear of his erstwhile team members.
Tomiko drew in upon herself, all centered in the blind eye in her belly, in the center of her being; and outside that there was nothing but the fear.
She raised her head; slowly unclenched her hands. She sat up straight. The night was dark, and stars shone over the forest. There was nothing else.
It was Eskwana's voice. She snapped on the interior lights and saw the engineer lying curled up asleep, his hand half over his mouth.
The mouth opened and spoke. "All well," it said.
"All well," said the soft voice from Eskwana's mouth.
"Where are you?"
A wind was rising. "I'll stay here," the soft voice said.
"You can't stay—"
"You'd be alone, Osden!"
"Listen." The voice was fainter, slurred, as if lost in the sound of the wind. "Listen. I wish you well."
The Wikipedia page for the story has a more complete plot summary.