This sounds like the 2007 novelette "The Merchant and the Alchemist's Gate" by Ted Chiang; the story is anthologized in his 2019 Exhalation: Stories. Several of Chiang's stories confront the distinction between free-will versus determinism, with Chiang expressing a consistent optimism that meaning in life still inheres absent free choice (a theme strongly present in the 2016 film Arrival, for example, and his 1998 "Story of Your Life" on which the film is based).
In the story "The Merchant and the Alchemist's Gate"—which indeed takes place in the Baghdad of the Abbassid Caliphate, and involves older and younger versions of a few characters—there are portals (made available by the alchemist character) to the future and to the past. I would argue that, in contrast to your wondering if "there really is no point" (which of course is a valid take), that one of the points made by Chiang regarding the fabric merchant character is that, given both the changing nature of memory, and the situatedness of our values and existential experiences in where we are in the present moment, that a foregone conclusion (like an unchangeable past/future even with time travel), is still worthwhile, meaningful, and even surprising.
Here are the covers of the novelette, and of the anthology, respectively:
There seem to be several podcast renditions of this story, although the one at starshipsofa was apparently removed at the request of the author.
