Reincarnation of the Avatar is definitely a thing
In Book 3's "The Avatar and the Fire Lord", Avatar Roku shared with Aang a memory of his past, which involved his death. Immediately following that, Aang saw himself as a newborn baby. From the Avatar Wiki transcript:
With the rock avalanche just moments away from swallowing Roku, Fang flies down, and wraps himself around Roku. The wave of ash and rubble cover the two completely, killing both of them. Fade to a bright light, as the camera pans to show an arched window frame. A baby's cries can be heard. The camera pans down to show a smiling born infant.
Aang: Who's ... wait, that's me, isn't it? [Cut to a long shot of three Air Nomads. A nun holds up the baby to the light while two people, presumably the infant's parents, stand in the shadows.]
See also: Are other human spirits reincarnated in the world of Avatar: The Last Airbender?
Raava is permanently bonded to the Avatar and stays with them throughout each lifetime
In The Legend of Korra "Beginnings, Part 2", Raava permanently bonds with Avatar Wan. From the Avatar Wiki transcript
While still static, Wan lights up briefly, showing Raava's form inside his chest. He puts his right hand behind him, causing the energy of the southern spirit portal during Harmonic Convergence to flow though him, basking him in blue light, as he permanently merges his spirit with Raava's. […]
Raava: We are bonded forever.
An old Avatar Wan laments at the fact that he couldn't defeat evil in his lifetime. Raava responds:
Don't worry. We will be together for all of your lifetimes. And we will never give up.
We see that the Avatar can be identified from an early age
The monks knew Aang was the Avatar since he was a child:
You chose them from among thousands of toys, Aang. The toys you picked were the four Avatar relics. These items belonged to Avatars past. Your own past lives.
—Monk Tashi, Book One Episode 12 The Storm
Korra too was identified as the Avatar when she was a few years old, but unlike most Avatars, it was blatantly obvious since she could bend multiple elements from that age.
So I think it's safe to say that Avatars are who they are from their birth.