The Potters were protected by the Fidelius Charm, they had no need of the Invisibility Cloak, and they surely didn't expect Petter Pettigrew to turn to Voldemort and tell him the secret.
Besides that, the Invisibility Cloak wouldn't have protected them, as Harry wisely states:
‘The Cloak wouldn’t have helped them survive, though,’ Harry said
quickly. ‘Voldemort knew where my mum and dad were. The Cloak couldn’t
have made them curse-proof.’
‘True,’ sighed Dumbledore. ‘True.’
Rowling, J.K.. Harry Potter and the Deathly Hallows (p. 584).
Pottermore Publishing. Edición de Kindle.
Voldemort knew where they were. He could've performed a simple Homenum Revelio to detect the Potters, and then blast them away with whatever curse he wanted. He could've simply blasted down the house with the Potters inside.
He could've done a lot of things, even if James still had his Invisibility Cloak.