According to a few sources the FX were done by matte paintings until 3D software such as Maya and LightWave became more practical to use. Which was around the time Voyager graced the screen.
Unless of course you were watching a blue ray remaster.
"More static vistas such as long views of landscapes, cities and space vistas, planets or other celestial bodies in particular, were traditionally done as matte paintings, predominantly (though, on rarer occasions, such elements were sometimes executed as maquettes)."
"However, the advent of paint computer programs like Microsoft Paint and Adobe Photoshop (though far more advanced computer software is used in the motion picture business, as they are able to manipulate 2D images into 3D imagery) meant that many matte artists traded their glass canvasses and brushes for a computer mouse and screen. John Grower stated in this regard to the main title sequence of Voyager, in the section where the namesake starship flies over an icy moon, "This foreground moon was actually a matte painting done by Craig Mullins, but then we took the matte painting and mapped it onto three-dimensional geometry, so it became a 3-D matte painting"
According to io9.com (who was quoting Star Trek TNG Technical Manual) a mylar pom pom once served as a nebula.
Thewertzone has this to say regarding the blueray remaster.
What was not anticipated was that the a lot of the original matte paintings, particularly of the planets that the Enterpriseorbits, had gone missing in the intervening time and the surviving low-res bitmaps were not of any use. So almost every single planet in the series (barring the one in 11001001, which is a high-res matte painting of Earth taken from The Search for Spock) has been replaced by a highly-detailed 3D model, based on the corresponding matte painting but with far more detail and, noticeably, animation (i.e. the planets spin as the Enterprise orbits them). This also allowed the effects team to fix a few instances of blatantly the same planet being re-used by altering terrain details and a few colours.