According to the Fellowship of the Ring script at Council of Elrond, in the film Gandalf first tries:
Annon Edhellen, edro hi ammen! (Gate of the Elves, open now for me!)
Fennas Nogothrim, lasto beth lammen. (Doorway of the Dwarf-folk, listen to the word of my tongue.)
Ando Eldarinwa…a lasta quettanya, Fenda Casarinwa… (Gate of Elves…listen to my word, Threshold of Dwarves…)
In the book, Gandalf first tries:
Annon edhellen, edro hi ammen!
Fennas nogothrim, lasto beth lammen!
Many times he repeated these words in different order, or varied them. Then he tried other
spells, one after another, speaking now faster and louder, now soft and slow. Then he spoke many single words of Elvish speech. Nothing happened. The cliff towered into the night, the countless stars were kindled, the wind blew cold, and the doors stood fast.
Again Gandalf approached the wall, and lifting up his arms he spoke in tones of command and rising wrath. Edro, edro! he cried, and struck the rock with his staff. Open, open! he shouted, and followed it with the same command in every language that had ever been spoken in the West of Middle-earth. Then he threw his staff on the ground, and sat down in silence.
It should probably be mentioned that in the book, 'Mellon' is Gandalf's idea (inspired by something Merry said) whereas in the film 'Mellon' is Frodo's idea.