I don’t think it had any special name; it was just “the galaxy”. Several of the Star Trek wikis refer to the Milky Way, but I can’t find any reference to it being named as such in an episode; I suspect this is for out-of-universe convenience.
The Star Trek Expanded Universe wiki page concurs:
The Milky Way Galaxy (translated from Latin Via Lactea, derived from the Greek Γαλαξίας (Galaxias), often referred to as simply “the galaxy”, is a large barred spiral galaxy in the Local Group.
Memory Alpha doesn’t say anything on the subject.
Memory Beta also mentions the name “home galaxy”, from a presumably non-canon game. Make of that what you will.
The Milky Way is a spiral Galaxy in which most of recorded history takes place. The name “The Milky Way” comes from Earth; the galaxy is also known as the “home galaxy.” (TNG video game: Echoes from the Past)
In short, I think it was just “the galaxy”, and the name “Milky Way” was ignored in the Star Trek universe.