They are going to Castle Black, but it's slightly round-about because they are part of a two-pronged attack and have to circle around behind.
From the map below you can see that they climbed the wall roughly between Greyguard and Stonedoor. Once over the wall, they'll need to move away from it for two reasons.
- To avoid any Night's Watch traveling between The Shadow Tower and Castle Black.
- Because they need to live off the land as they wait for the signal to attack.
Indeed, the attack on Castle Black comes a full 13 episodes later, no mean amount of time in-universe. So the wildings travel south to a slightly more habitable climate to wait for the attack.
I think the amount of green they see is more a byproduct of the magic in The Wall than an indication of how far they traveled. The Gift is only 50 leagues (roughly two days walk) from Wall to border, so they couldn't have gone that far, even if they had traveled to the very edge. We know that they appeared in Moletown, so odds are they spent most of the journey within 5-10 leagues of the wall.
From the Beyond the Wall map in Lands of Ice and Fire