The implication was that Rey was somehow blocking him through her own efforts to move the hilt. Note his surprise that an object so small doesn't come on command.
The exertion actually causes him physical pain which in turn weakens his Force abilities to the point that Rey can overpower him and call the lightsaber to her.
He retracted his blade, turned, and reached out a hand for Finn’s lightsaber.
The hilt wobbled in the snow. Pulling it through the Force proved more strenuous than it should. The pain of Ren’s wound clouded his concentration.
Spoilers for Last Jedi:
Obviously when he's in Snoke's chamber there's no opposing force, at least not initially. When he later contests Rey for the hilt (this time without an injury to distract him) they end up rending it in twain.
