Spock is written as ineffective when confronted with his charismatic, rule-breaking half-brother, Sybok
Spock is humiliated twice in confrontations with Sybok, making him look naive, vacillating, and ineffective.
You are under arrest for violating
seventeen counts of Neutral Zone
Sybok is incredulous. There must be a hundred guns
pointed at Spock's heart. Sybok can't help it. He
chucles. The chuckle becomes a full rich laugh. The
soldiers join in and soon their laughter is ringing
He can't even handle Sybok when armed:
Spock raises the weapon as Sybok advances.
Spock -- shoot him!
But Spock can't shoot. The emotional price is too great.
He lowers the weapon in shame and defeat. Sybok takes
it from him with a smile.
We are made to think that it is unclear where Spock's loyalties lie:
KIRK: Spock! My only concern is getting the ship back.
When that's done and Sybok is in here, you can debate
Sha Ka Ree until you're green in the face. ...Until then,
you're either with me or you're not.
SPOCK: I am here, Captain.
KIRK: That's a little vague, Spock.
This may have been related to the early idea behind the script that everyone would turn against Kirk (via Memory Alpha):
William Shatner's first outline for this film was entitled "An Act of Love" and, according to William Shatner's Star Trek Movie Memories, would have been a much darker tale and would have seen the first true falling out between Kirk and Spock and McCoy. Also, Spock and McCoy would also have joined with Sybok, leaving Kirk alone. This was changed when Nimoy absolutely refused to play that, stating that there was no chance whatsoever that Spock would ever turn on Kirk, especially after what Kirk risked and sacrificed for Spock in Star Trek III. Director Shatner talked to Nimoy, attempting to change his mind, but Nimoy was firm in believing that pain or no pain, brother or no brother, Spock would not betray Captain Kirk. Shatner eventually conceded and had the script adjusted.