All source based on Legend of Korra - Beginnings Part 1 & 2:
Before the first Avatar - the humans did live together - though they did not possess the ability to bend. When Vaatu broke the barriers between the mortal and spirit worlds - humans had to break apart and go into hiding to protect themselves from the spirits. The lion turtles offered shelters to humans and villages were built around the lion turtles' backs. These lion turtles possessed the ability to bend and shared that art with humans when they needed to go among the spirit wilds to gather food - but usually took the power back once the humans returned home.
One such human - Wan - refused to give back his firebending ability and instead went on travels to find the other lion turtle cities. He eventually mastered all four elements and became the first Avatar. He was also crucial in releasing Vaatu and banishing the spirits back to the spirit world.
Now - with the humans no longer needing protection - the lion turtles encouraged the humans to live on their own. However - since the villages and the culture that the humans had built when under the protection of the lion turtles - the humans were comfortable in their own territories and did not bother living together again. The comfort factor may be added by the fact that they chose to stay in areas where their bending will be more in touch with the elements.
From various accounts of both Aang and Iroh among others - it looks like they did live in harmony and did travel frequently among other territories. The reason they probably chose to stay isolated was probably because of their cultural differences.