I remember a short story where two rocks are talking to each other, and during the conversation (on their timescale) a giant city rises from nothing, is built right up to the edge of the rocks, and then crumbles before them.
Could it be The Great Slow Kings, by Roger Zelazny? It's not a perfect match, but pretty similar - two kings of an alien species have a conversation, but their minds work on huge timescales, so by the time they make any decision, the civilization about which they debated has risen and fallen.
I think you are remembering a short story contained within the slightly longer (but still short) children's book Mouse Soup by Arnold Lobel. Mouse Soup is about a mouse captured by a weasel. To delay being turned into the weasel's lunch, the mouse tells 4 stories he claims will show the weasel ways to make his mouse soup taste better.
One of the stories told is "Two Large Stones". Two stones are situated near each other on one side of a hill. They wonder what is on the other side of the hill. At various times they ask animals that come near to look on the other side of the hill, then return and tell the stones what they have seen. The animals give answers in line with what that kind of animal would pay attention to most, leading the stones to wonder which animal was telling the truth.
The other short stories contained within Mouse Soup are "The Bees and the Mud", "The Crickets", and "The Thornbush".
Another possible contender is the short story "Faldum" written in 1916 by the German author Herman Hesse. It appears in the collection "Strange News from Another Star". In it a musician is transformed by a wish into a mountain. There isn't exactly a conversation, but the rise and fall of a city is seen by the mountain and it fits your description.