I'm looking for a novel from the 70's (pretty sure). I believe it starts off with the court martial of a starship captain. The story lays out the reasons for the court martial. It turns out the ship was sent out to visit established colonies and just show the flag. When they arrive at one colony they detect a ship vanishing into hyperspace. They visit the colony and everybody has vanished. After a search some colonists are found. Only they've been hideously modified and are hiding in a cave. The captain sends everyone else back to the ship and talks to the colonists. They beg him to kill them, so he does. That's the reason for the court martial. The truth comes out and he is acquitted.
There was a second book where a starship arrives at another colony where the changes are just starting to take effect. The colonists are giving birth to monsters. A female crewmember gets pregnant and gives birth to a monster.
That's about all I remember. Starfleet headquarters was in San Francisco, I think.