Let's imagine a situation where Jack and Jill invent mathematics.
Jack has an apple. Jill has an apple. Jack gives Jill his apple, because reasons. They reference this event as "Jack and Jill at The Hill. Jack, his stomach full. Jill, her hunger great." Perhaps they even reference Temba as they talk to each other, who knows. They shorten this story to "Jack and Jill, lunchtime at The Hill."
One day Jill needs to defend herself against a vampire, because reasons. She has one stick, but requires another to form a cross, which always repels vampires. She references their earlier story to Jack -- "LUNCHTIME AT THE HILL LUNCHTIME AT THE HILL" and points at the stick near Jack. He tosses it to her. She drives off the vampire. Then she gives the stick back.
This situation can spawn many stories -- one of them being "that time we realized that it's helpful to say 'I need two of this thing'" and gets its own label. "Jill, preparing a weapon at The Hill." We've seen Dathon step through a story before -- he does so in an effort to walk Picard through the narrative of Darmok and Jalad -- so we know that Tamarians are not ignorant to the concept of telling a story one scene at a time. Therefore we know they are capable of introspection; a story has a beginning, middle, and end, and each part of it can be used to service communication of specific events.
So this aspect of the story represents, conceptually, 'one thing needing one more thing' or 'one plus one'. And perhaps "Jill, retiring her weapon" can be giving the stick back; returning what's rightfully someone else's; it can also be 'one minus one'.
From these humble beginnings spring the basics of mathematics. The necessity of communicating "two plus two" comes after enough repetition and refinement of the original process of doing so, as in the progressive story of Jack and Jill at the Hill.
The nature of the language of the Children of Tama encapsulates referencing complex situations to convey a message. There is little difference between describing such a rich series of events, or a drudgery of wasted effort, and a complex mathematical procedure.
For example -- I could say "Descartes, his line on a graph." and we could (potentially) understand the process as a reference to y = mx + b. The math needed to do the work would have been conveyed previously through other educational process; just as we cannot start children with slope-intercept form in kindergarten, so too would a member of the Children of Tama require reference. But, as illustrated, it only takes one instance of "that time we did that thing" to convey basic math (1 + 1, 1 - 1) and the rest is simply an issue of repeated exposure and history. The story of Isaac Newton and the apple allegedly gave birth to his theory of gravity; if our culture's stories were adapted to speak to the Tamarians, this story could be referenced during conversations of physics. "Newton, when the apple fell."
We assume the precision of our conversation is in the specificity of each word, or each sentence, but really our language is replete with its own built-in imagery -- we just don't lean on it quite as Tamarians do.
None of this is canon of course, but it is simply meant to describe that it is not impossible or beyond simple scrutiny to see that it's just a question of linguistic building blocks. (Besides, you didn't ask for canon, you asked "how could it be so?" and this is one such method.)