My guess is that it vents heat by expelling superheated gases (not plasma). Everyone is right that the only way heat can escape a body in space is by radiation, so an "exhaust port" designed to expel pure heat the way a car's exhaust would wouldn't work very well.
HOWEVER, that's not the only way to remove heat while in a vacuum. A very effective way to cool heated surfaces such as heatsinks is to spray a liquid onto that surface, forcing conduction of the heat away from said surface, and then expelling the now-heated liquid or gas into the void. As the main source of heat in the Death Star would be its reactor core, should that core overheat, you can cool the entire system by simply expelling some of the coolant and replenishing it.
The original novel by Lucas states, as an extension to the dialogue heard in the movie, something along the lines of "the shaft would be rendered useless by particle shielding; however, it is completely ray-shielded". This further increases my confidence that the "thermal exhaust" consists primarily of matter and not energy. And, we know there's a main port that usually handles most of the station's cooling needs, but this port can likely be particle-shielded to protect it in an attack, leaving the auxiliary port to handle the cooling needs.