The short answer... they didn't.
The long answer (which does't include SFX beyond "hair and make up").. they chose actors to fit the characters for Kirk (Corn fed Iowa farm boy), Spock (Tall dark & impassive) etc. Then they go through hair and makeup, this makes them superficially similar to the actors you know and love.
Then the real magic happens. The actors play their parts, and embody the characters.
For example, the actor Karl Urban doesn't look much like the actor DeForest Kelley
But Urban's portrayal of Doctor Leonard "Bones" McCoy is incredibly convincing, and arguably the best "new for old" character in the movie.
Also be aware that you were expecting to see those characters, so your imagination helped complete the illusion that the new actors were so very similar to the old.
Also the movie's Wikipedia page has examples showing how the characters were interpreted differently by the new actors
Pine felt he had to show Kirk's "humor, arrogance and decisiveness,"
but not Shatner's speech pattern, which would have bordered on
imitation
and
Like Pine, Urban said of taking on the role that "it is a case of not
doing some sort of facsimile or carbon copy, but really taking the
very essence of what DeForest Kelley has done and honoring that and
bringing something new to the table".
There's lots more on the movie's Wikipedia page.