Isn't Luke able to defeat the rancor using the Force (by means of choking, knocking, or any other telekinetic force attack using the Force), or it does not work on these monsters?
There are a few reasons;
Luke isn't very powerful at this point. He lacks the ability to perform a Force choke and certainly can't do it while fighting for his life. Although he originally was stated (in the first novelisation) to have Force-choked the Gamorrean guards, the latest canon novelisation indicates that he actually mind-tricked them into thinking they were choking, a much easier skill to master and one consistent with the light side of the Force.
Force Choke is (generally) used by dark side Force users given that it's unnecessarily violent.
This ability perfectly exemplifies how acolytes of the dark side can twist the Force's powers to their sinister whims. Often employing a simple clawlike hand gesture to focus the attack, a practitioner of the dark side is able to use this ability to remotely strangle anyone who opposes them. Some use this corrupt form of telekinesis merely as a method of punishment and control while others take the power much further and crush the life out of their foes. Although quite effective, this ability is rooted in fear and designed for pain, making it a power that a Jedi rarely calls upon.
Luke had no animosity toward the Rancor. His goal was to escape the enclosure or, at worst, to put the creature out of its misery.
It was not an evil beast, that much was clear. Had it been purely malicious, its wickedness could easily have been turned on itself—for pure evil, Ben had said, was always self-destructive in the end. But this monster wasn’t bad—merely dumb and mistreated. Hungry and in pain, it lashed out at whatever came near. For Luke to have looked on that as evil would only have been a projection of Luke’s own darker aspects—it would have been false, and it certainly wouldn’t have helped him out of this situation.
Force choke would probably not be as effective as one would like against an attacking beast.
I used to have a large dog that was very hyper, not super smart, and very dominant. Generally I'd tell dog owners not to do this, but for her I was forced to walk her on a choke chain. (This was recommended to me by a professional trainer that spent a whole day working with her. Do not do this with your own dog!)
The choking was somewhat effective to get her attention when she ran to the end of her leash, but not nearly as much as you might imagine. There was one particular time I remember her really wanting to go mob another dog so badly that she was up on her hind legs, straining against the chain, with her happy tongue lolling out of the side of her mouth slowly turning blue.
A rancor is not only in this same category of being very aggressive and dominant and not particularly smart, but seems meant to be the epitome of that kind of creature. So an enraged rancor is likely to react roughly the same way to being choked that my dog did that day: not at all. In fact its quite likely he wouldn't stop or slow down until rendered unconscious through actual asphyxiation. For the situation Luke was in, with it charging right at him, he didn't have that kind of time.
Because few people are able to think of everything in the excitement of a dangerous situation, it quite possibly didn't occur to him to try.
On top of this, Luke is not a Star Wars fan and doesn't have the luxury of reviewing the films, looking for a way to have done better, so he had to rely on his experiences and training up to that point. (Otherwise, he probably wouldn't have tried killing Ben Solo in his sleep, and also probably would not have let Leia kiss him. And so on.)
The reason for the snark: It seems to be a common thing for people to claim to have found a plot hole in the behavior of a character in a given situation that character did not do the most logical thing. In some cases what seems obvious to the audience is not necessarily obvious to the characters.
Too many unknown variables to know for certain.
Others have already mentioned that Luke was far from calm, and was reacting to a very dangerous situation, so he might just have not thought about it.
The other unknown is how the Force itself affects the Rancor. Throughout the movies we've seen different creatures (and even different people) have different levels of resistance. Just before falling into the pit, Jabba himself had resisted Luke's attempts at mind tricking him, and subsequent movies present other creatures that are resistant (it takes some effort for Anakin to subdue one of the creatures in the Geonosian arena) to outright possessing some form of immunity (Watto's species is immune to mind tricks).
My personal interpretation is that the ability to influence living things is always in flux depending on the circumstances, hence why you don't see Jedi using their powers constantly (it would've been useful to mind control the Ewoks, for example), and, furthermore, that large creatures, having more "life" by virtue of their size, are probably more resistant.
This interpretation might explain why most effects we see on the movies are usually focused on inanimate objects (like telekinesis), while directly influencing living things is somewhat rarer (in the original trilogy specially, as both the force choke and force lightning were presented as proof of Vader and Palpatine being exceptionally powerful).
But it's just speculation.
- It is a dark force power - therefore, as a light Jedi, Luke wouldn't know it. (You might as well ask why he didn't just shoot back with force lightning at the Emperor.)
- Not every power is known by every force user either. Did anyone BUT Palpatine ever use force lightning? Did anyone BUT Vader use force choke in the movies? Luke never demonstrated knowing this ability, so why'd you even assume that he has it?
- Let's step outside the video game logic for a moment (though, even video game abilities work with limitations - an insta-kill spell for NPCs and mobs would probably not be possible to use on boss monsters) and consider why, and how, this force application works. It is actually a very minor display of telekinesis (apart from the fact that Vader can apparently use it through a screen, from a ship lightyears away), very little force in a tiny spot, so it wouldn't require as much force power as rather precision, so that you don't accidentally squeeze just a bit too hard and kill the target before it could be suitably intimidated. It'd require very fine tuned telekinesis practice and also familiarity with human biology. No telling if that still would work in the case of a MUCH larger creature with unfamiliar alien biology. I don't expect ever to see Vader choke any nonhumanoids or space monsters in any version. (By the way "force choke" "force jump" "force push" are likely just fancy names, denoting the technique and not the source, all being really the same ability: force telekinesis ie. moving shxt around with your mind.)
In the original trilogy, and in earlier depictions of the Star Wars universe, the Force was a much more subtle thing than what we witnessed in the
remake new trilogy, where people can instantly develop ultra-powerful new abilities on the spot as the plot requires it, can rip starships out of the sky, lift a gazillion tonnes of rock, and teleport around across huge distances.
The Force was imagined as something like magic in the Lord of the Rings. You don't see Gandalf flying around tossing fireballs left and right, ripping mountains apart and defeating huge armies all alone. You don't see Obi-Wan in the original trilogy collapsing the ground beneath the Tusken raiders or force-throwing them across the desert. You don't see Luke just throwing the Rancor to the ceiling, then smashing Jabba's palace to bits with a Force-Earthquake. The Force is a subtle thing, you can use it for small and careful feats. The Force guides you, and you can subtly manipulate the real world through it, but in order to win you need grit and effort. It's one thing for one of the most powerful Force-wielders to crush a human's throat, a feat a regular human can do one-handed given physical contact, but a huge monster has too thick skin and too sturdy throat to crush it with your hand, so force-choking it would require a lot more power.