Look at this source from a Pottermore article by JK Rowling (the full article can be found here):

The wizarding family of Potters descends from the twelfth-century wizard Linfred of Stinchcombe, a locally well-beloved and eccentric man, whose nickname, ‘the Potterer’, became corrupted in time to ‘Potter’.

Linfred’s eldest son, Hardwin, married a beautiful young witch by the name of Iolanthe Peverell, who came from the village of Godric’s Hollow. She was the granddaughter of Ignotus Peverell. In the absence of male heirs, she, the eldest of her generation, had inherited her grandfather’s invisibility cloak. It was, Iolanthe explained to Hardwin, a tradition in her family that the possession of this cloak remained a secret, and her new husband respected her wishes. From this time on, the cloak was handed down to the eldest in each new generation.

The Potters continued to marry their neighbours, occasionally Muggles, and to live in the West of England, for several generations, each one adding to the family coffers by their hard work and, it must be said, by the quiet brand of ingenuity that had characterised their forebear, Linfred.

Henry Potter ... , who was a direct descendant of Hardwin and Iolanthe, ... served on the Wizengamot from 1913-1921.

Henry’s son was called Fleamont Potter. ... It was Fleamont who took the family gold and quadrupled it, by creating magical Sleekeazy’s Hair Potion (‘two drops tames even the most bothersome barnet’). He sold the company at a vast profit when he retired, but no amount of riches could compensate him or his wife Euphemia for their childlessness. They had quite given up hope of a son or daughter when, to their shock and surprise, Euphemia found that she was pregnant and their beloved boy, James, was born.

Fleamont and Euphemia lived long enough to see James marry a Muggle-born girl called Lily Evans, but not to meet their grandson, Harry. Dragon pox carried them off within days of each other, due to their advanced age, and James Potter then inherited Ignotus Peverell’s Invisibility Cloak.

From the above, it is quite clear how the Cloak came into James' possesion.

Look at this excerpt from a Pottermore article by JK Rowling (the full article can be found here):

The wizarding family of Potters descends from the twelfth-century wizard Linfred of Stinchcombe, a locally well-beloved and eccentric man, whose nickname, ‘the Potterer’, became corrupted in time to ‘Potter’.

Linfred’s eldest son, Hardwin, married a beautiful young witch by the name of Iolanthe Peverell, who came from the village of Godric’s Hollow. She was the granddaughter of Ignotus Peverell. In the absence of male heirs, she, the eldest of her generation, had inherited her grandfather’s invisibility cloak. It was, Iolanthe explained to Hardwin, a tradition in her family that the possession of this cloak remained a secret, and her new husband respected her wishes. From this time on, the cloak was handed down to the eldest in each new generation.

The Potters continued to marry their neighbours, occasionally Muggles, and to live in the West of England, for several generations, each one adding to the family coffers by their hard work and, it must be said, by the quiet brand of ingenuity that had characterised their forebear, Linfred.

Henry Potter ... , who was a direct descendant of Hardwin and Iolanthe, ... served on the Wizengamot from 1913-1921.

Henry’s son was called Fleamont Potter. ... It was Fleamont who took the family gold and quadrupled it, by creating magical Sleekeazy’s Hair Potion (‘two drops tames even the most bothersome barnet’). He sold the company at a vast profit when he retired, but no amount of riches could compensate him or his wife Euphemia for their childlessness. They had quite given up hope of a son or daughter when, to their shock and surprise, Euphemia found that she was pregnant and their beloved boy, James, was born.

Fleamont and Euphemia lived long enough to see James marry a Muggle-born girl called Lily Evans, but not to meet their grandson, Harry. Dragon pox carried them off within days of each other, due to their advanced age, and James Potter then inherited Ignotus Peverell’s Invisibility Cloak.

From the above, it is quite clear how the Cloak came into James' possesion.