My memory on this one is rather fuzzy. Hopefully, I'm recalling enough of the story correctly that someone will recognize it.
As stated in the question title, in this tale, it has become possible to place items, including organic matter, inside every day solid objects for safe-keeping. Said objects are in a form of stasis, IIRC, which allows organic matter to be kept indefinitely without degrading. Most any object can be used (again, IIRC). Earrings, a lamp, etc.
The protagonist is a thief by trade, and among his conquests was a pricey vase or the like from a female friend of his. I do not recall if she was his lover/girlfriend or not. I'm fairly certain her name was "Beth." We do not find out about what makes the item so special until nearly the end of the story.
I am fuzzy about the details of how the story progresses, but the gist of it is, the protagonist gets caught up in a double-cross. One of the people he burgled ended up dead, and he is believed to be the murderer. He's caught, convicted, and sentenced to death.
The final paragraphs of the story have our protagonist sitting on death row in prison. We learn that the entire chain of events that lead to his frame was masterminded by Beth. She set him up as revenge because the item he stole (and subsequently fenced/sold/is otherwise forever lost) contained her ovaries. I think the reason she stored them there was that she wanted to keep them in stasis until she was older and ready to have children, though there might be another factor at work as well.
The story closes with the protagonist lamenting that he was unable to get Beth to forgive him, and rhetorically asks "she won't really let me die - will she?"
I'm certain this one came out no earlier than the late 2000's. I also definitely read it in a sci-fi magazine, not an anthology. The title might be a single word, though I am not certain about that last point.