How does the dagger work in Prince of Persia? If the holder goes to the past to change things what becomes of that time/world/characters which went through that time? Is it illusion? If it was illusion, then everything becomes illusions for everyone (since the holder can be anyone) which is impossible.
