I believe that you are looking for Destination: Void (1965), by Frank Herbert.
In the future, mankind has tried to develop artificial intelligence, succeeding only once, and then disastrously. A transmission from the project site on an island in the Puget Sound, "rogue consciousness!", was followed by slaughter and destruction, culminating in the island vanishing from the face of the earth.
The current project is being run on the moon, and the book tells the story of the seventh attempt in a series of experiments to create artificial consciousness. For each attempt, the scientists raise a group of clones. These clones are kept isolated and raised to believe that they will be the crew of a spaceship that will colonize a planet in the Tau Ceti solar system (Tau Ceti has no habitable planet; its choice—should they manage to reach it—is part of the planned frustration of the crew). The spaceship will take hundreds of years to reach the system and the crew will spend most of their time in hibernation. Along with the crew of six, the ship carries thousands of other clones in hibernation, intended to populate the new colony and, if necessary, provide replacements for any crew members who die along the way.
The crew are just caretakers: the ship is controlled by a disembodied human brain, called "Organic Mental Core" or "OMC", that runs the complex operations of the vessel and keeps it moving in space. But the first two OMCs (Myrtle and Little Joe) become catatonic, while the third OMC goes insane and kills two of the umbilicus crew members. The crew is left with only one choice: to build an artificial consciousness that will enable the ship to continue. The crew knows that if they attempt to turn back they will be ordered to abort (self-destruct).