Voldemort's face had not fully changed when he had the work appointment with Dumbledore. However, it had minor changes. This was probably around about the time it started to change, the year then was 1956, or something around 1956. This is a bit from Half-Blood Prince, Chapter 20, Lord Voldemort's Request, the memory of his appointment with Dumbledore.
Harry let out a hastily stifled gasp. Voldemort had entered the room. His features were not those Harry had seen emerge from the great stone cauldron almost two years before; they were not as snakelike, the eyes were not yet scarlet, the face not yet masklike, and yet he was no longer handsome Tom Riddle. It was as though his features had been burned and blurred; they were waxy and oddly distorted, and the whites of the eyes now had a permanently bloody look, though the pupils were not yet the slits that Harry knew they would become. He was wearing a long black cloak and his face was as pale as the snow glistening on his shoulders.
But notice this line:
That Harry knew they would become.
This line shows that Voldemort's face has not all changed, only portions. It keeps on changing for many more years to come.
But how does his face change?
Well, I disagree, and agree, with the above posts. I think, sorry, not think, I am 95% sure, that his face changes because of his Horcruxes. In the meeting with Dumbledore, he had already made five Horcruxes. His face had begun to change because of this, and, like I keep saying over and over again, it kept changing for many more years to come. But from the gap between 1956 and 1981, the year his "first death" occurred, his face had not become the one we see when Voldemort is reborn in the graveyard in Harry Potter and the Goblet of Fire. Yes, by 1981, it had become very different from the one in the meeting with Dumbledore, but, like I mentioned a few lines above, not the one we see in the graveyard.
You see, by the time his first death occurs, he has already made six Horcruxes, the new Horcrux is Harry. But it is still not the face we see in the graveyard, but at the same time, it is not the one from the meeting. No one can be sure of this, since we never see his face when he comes to try and kill Harry in 1981. But I am 95% sure I have got my facts right. His face stays the unknown face until early 1994, in the Riddle House. When he is inside the Riddle House, he creates the seventh Horcrux, which is his snake, Nagini.
When Voldemort turns Nagini into a Horcrux, he get some of the snake-like features Nagini has, and his face changes once more, into the face we see in the graveyard, and the face we see in all of the events after the graveyard. Done!