Note what happened the first time that Angier used the machine:
He kept a loaded gun handy, nearly within reach from his place inside the machine. When he saw his duplicate, he immediately, almost instinctively, moved to kill the duplicate.
My conclusion based only on this action would be that Angier is afraid of the consequences of what his machine does. He also states as much at the conclusion of the film:
"Do you want to see what it cost me?" ... "Let me show you. It took courage to climb into that machine every night. Not knowing if I'd be the Prestige, or the man in the box."
(This quote is taken from a copy of the script, which I don't know the absolute accuracy of.)
This line, particularly the statement that "it took courage", exemplifies Angier's existential fear regarding his situation. He's afraid of what this seemingly supernatural machine implies about humans, or living things, or reality itself:
He knows now that a living human being can be duplicated exactly, retaining all of its thoughts, feelings, and memories. When the first duplicate appears, it's clearly startled. Angier #2 cried out "No, wait! I'm the-" before Angier #1 shot him. I think we can assume that Angier #2 was about to say that he's the "real" or original Angier. He believed himself to be the one who entered the machine, and did not expect to find himself across the room.
If the two are physically identical, and share the same memories up to the moment that the machine was activated, there's no way to distinguish them. This shakes Angier #1's own belief that he is the original, inviolable being that has lived its life as Robert Angier up to this point. He can never know if he's the same physical being who entered the machine, or a new Angier who was created by the machine. He is afraid that he is not the "real" Robert Angier.
The issues this raises about the nature of thought are far beyond the scope of this question. Suffice it to say: we believe we continue through life as the same person because of our unique memories. If these memories could be duplicated, it would create doubt that we are truly the individuals we believe ourselves to be.
A living, breathing reminder of that doubt was too much for Angier to bear. He could never banish that doubt from his mind - as represented by both his final statements and his warehouse full of drowned duplicates - but he was afraid to face it fully by allowing a duplicate to live.