Another question triggered my memory of a different story, this one short, likely Omni in the 1980s.
A former child prodigy and now professional mathematician works at a sealed think-tank, I think in the deserts in the US southwest. He begins getting visits from a woman and mistakenly mentions this to the head of security, who is an almost twin for the person he thought he was talking to.
The next time she visits the security team breaks in to his room find them gone, leaving behind only a sort of wooden idol, broken in half and sickly green along the break. They tear the place apart to find them, finding only long forgotten microphones.
The woman was actually his sister, the number 6, and he was really 5. Both returned to the Cartesian plane of pure math. That might sound dumb, but it really was a pretty engaging story.