The main character (male) is called to a lab built on the ocean floor that is atop some sort of vault that is in need of the main character's expertise in figuring out the password to unlock it. The calculations always come up with impossible numbers like 0/0. Does this ring any bells? I believe the main character appears in other novels by the same author. Published within the last 10-15 years.
This distinctly reminds me of Deep Storm, by Lincoln Child, and is part of his Jeremy Logan series of books (the others being The Third Gate, The Forgotten Room, Full Wolf Moon).
The protagonist is requested to investigate a medical issue aboard a drilling platform in the Atlantic Ocean, but when they arrive they discover that the real issue is aboard a research station on the ocean floor.
The research station is investigating a mysterious source of transmissions coming from objects positioned above an anomaly buried deep below the station - the transmissions include a mathematical expression:
x = 1/0
Some of the ideas put forward by characters in the story posit that the anomaly is a library of knowledge that some alien civilisation has left there for humanity to discover, the transmissions being a coded message which needs to be deciphered to unlock it - other ideas are that the transmissions are a warning and the anomaly needs to be left alone.
While this is a Jeremy Logan book, he only appears tangentially in it - the main protagonist is a different character.
Could it be Sphere by Michael Crichton, perhaps?
Journey to the ocean floor - check
Puzzle how to open a mysterious object - check.
Impossible things found - check. (Example - impossible sea creatures such as shrimp with no digestive organs. I don't recall if 0/0 was part of it.)
Preview available on Google Books. See if it looks at all familiar.
At least a partial match to Peter Watts' Rifters series, which is about a group of genetically modified humans working in a deep-ocean lab/power plant that accidentally release a pandemic causing microorganism. The main character is Lenie Clarke who is blamed for the release and her name is subsequently taken by AIs as a name for a bunch of cyber attacks/viruses.
IIRC at one point in the series (I think in book 3 or 4). One or more of the main characters returns to the lab after being on the surface for some time to find that there has been a coup of sorts and that they are locked out/excluded and they need to hack their way back in.
Note that these books are freely available from Watts' site rifters.com and a number of other sites too without DRM.