If you're asking about the black hole and wormhole scenes, Kip Thorne and the others who worked on the renderings published a pair of papers on how they did it:
Gravitational Lensing by Spinning Black Holes in Astrophysics, and in the Movie Interstellar
Visualizing Interstellar's Wormhole
As mentioned in the first paper, they designed their own software called DNGR (Double Negative Gravitational Renderer):
For this, our team at Double Negative Visual Effects, in collaboration
with physicist Kip Thorne, developed a code called DNGR (Double
Negative Gravitational Renderer) to solve the equations for ray-bundle
(light-beam) propagation through the curved spacetime of a spinning
(Kerr) black hole, and to render IMAX-quality, rapidly changing
images. Our ray-bundle techniques were crucial for achieving
IMAX-quality smoothness without flickering; and they differ from
physicists’ image-generation techniques (which generally rely on
individual light rays rather than ray bundles), and also differ from
techniques previously used in the film industry’s CGI community.
And on p. 3-4 they write:
A primary goal of today’s state-of-the-art, astrophysical ray-tracing
codes (e.g., the Chan, Psaltis and O ̈zel’s massively parallel,
GPU-based code GRay ) is very fast throughput, measured, e.g., in
integration steps per second; the spatial smoothness of images has
been only a secondary concern. For our Interstellar work, by contrast,
a primary goal is smoothness of the images, so flickering is minimised
when objects move rapidly across an IMAX screen; fast throughput has
been only a secondary concern. With these different primary goals, in
our own code, called DNGR, we have been driven to employ a different
set of visualization techniques from those of the astrophysics
community—techniques based on propagation of ray bundles (light beams)
instead of discrete light rays, and on carefully designed spatial
filtering to smooth the overlaps of neighbouring beams; see Section 2
and Appendix A.
So, you can read the rest of the paper, especially Section 2 and Appendix A, for a lot more detail on how the software worked.
If you're asking about CG in other parts of the movie, Christopher Nolan tried to use physical models rather than CG as much as possible, he even claims "we made the decision early on not to do any green screens on the film" in this video. The ships were mostly done with physical models as discussed here (though 'digital models' are also mentioned, for example in the depressurization scene with debris), and this article says the robots were physical models in most scenes, only CG when shown running. Phantom42's answer has some info on what packages they used when they did have to use CG in scenes other than the black hole and wormhole.