If I remember right from one of the (fairly) recent anniversary editions of Dune (don't have it in front of me at the moment, think it was the 50th), the forward written by Frank Herbert says that he did tons and tons of research into economics, agriculture, religion, and more for writing Dune, and he did so much preparation for the novel that by the time he finished writing Dune he had the material and plans for writing the first three novels. I think the second was mostly finished when Dune was published, and the third was at least all planned out.
I don't remember it saying how the stories were originally published in magazines. He may not have started with the intent of writing a trilogy, but when he was done he knew it would be a trilogy and more.
I may try to update this answer with excerpts from the forward when I have a chance to check it.