This YouTube video has a shot of the rock you seem to be talking about:
I don't think it looks like much of anything (including a hand), but in any case, it's probably not supposed to represent anything, and is likely a natural feature already present on the location long before the cast and crew arrived. The scene was filmed on Skellig Michael1, a very small island off the southwest coast of Ireland.
From c. 600 CE to c. 1200 CE, Skellig Michael was the site of a monastery, but now it is inhabited only by millions of sea birds2 - and covered with abundant amounts of moss and grass; mile after mile of terrifying, steep, rickety, uneven, ancient staircases; and lots and lots of rocks.3
This last photo depicts the "beehive huts" in which the monks lived on Skellig Michael, some 1,500 years ago; I assume Luke was living in one of these structures
With so many rocks, it is inevitable that some of them look like stuff - our brains are hard-wired to recognize familiar patterns of things, even where they don't actually exist. The phenomenon is well-known, and is called "pareidolia".
This video shows the entire scene, albeit in a very small portion of the frame, at a weird angle, and with motion-sickness-inducing instability. In the movie, the camera (presumably mounted on a helicopter) begins to pan around the characters in a wide semi-circle; as it does so, the rock in question can be seen from many different angles. If it ever looks like a hand (and I still don't think it does), it does so only for an instant, before the camera moves a bit more and it looks like nothing/something other than a hand/just a rock again. I'm still chalking this one up to pareidolia.
1 Irish: Sceilig Mhichíl; Also known as Great Skellig, or in Irish, Sceilig Mhór.
2 In fact, the island is one of the most densely populated seabird rookeries in the world, and it has been designated as a nature preserve. This is why Irish environmentalist groups were furious about The Force Awakens being filmed there - that brief appearance is expected to bring a million tourists to the island next year, and there is considerable concern about how this traffic will affect the birds. The producers have heeded the uproar, and have constructed a lookalike set in a studio for use during the filming of Episode VIII.
I lived in Ireland for a while, and although I never visited the Skelligs, I have read a lot about them; I also follow Irish news, and was aware of the controversy before I saw the movie. I paid a lot of attention to the scene on Skellig Michael, and I was very curious to see just how much of the island and the monastery would be visible. The only clear onscreen evidence that the island was once inhabited are the staircases and a few glimpses of an enormous stone wall; both of these features are unmistakably man made.
3 In fact, the Irish often refer to Skellig Michael and its neighbor, Little Skellig (Irish: Sceilig Bheag), as "the Skellig Rocks", because when you get right down to it, that's all they really are - a couple of jagged rocks jutting out of the ocean, with some moss, grass, birds, bird poop (so much bird poop), and smaller rocks on top.