As the Ancient One explains to Bruce Banner
The Infinity stones create what you experience as the flow of time. Remove one stone and that flow splits.
and asks Banner to return them to their respective locations correctly.
Won't the destruction of the 6 stones impact the universe in the current time line?