Is Faux Olivia's baby with Peter still going to be born in the other world?
No, in the revised timeline the Peter Bishop that had a child with Faux Olivia drowned in Reiden Lake at age seven after an unsuccessful attempt by the alternate of his father to save him from a disease. It seems that due to his impact and connections in the original timeline to the people who still existed in the new timeline he was a paradox that was unable to be fully excised forcing September to erase him.
Peter and Faux Olivia never met to conceive that child and the child had very limited effect on the world and does not have the same resistance from erasure that Peter did. In "The End of All Things" September confirms that the son Peter was unaware of, Henry, ceased to exist when Peter activated the Bridge device.