I wish I had sources to cite for this beyond "I've been thinking about this, too." Fragile fan theory follows.
It’s unlikely Luke had many details to his plan. Given Anakin’s penchant for “make it up as we go,” it would make sense for Luke to have the same MO. Also like Anakin, he was probably counting on his abilities as a Jedi, and then filling in the gaps as they went.
Luke would have known he needed the whole group to go up against Jabba. (And really, which of them could he have told “no” and made it stick?) He also knew he needed to get Jabba out of the palace. Within his palace, Jabba was nearly untouchable. Out in the open, Luke would be able to (at the least) escape with the party intact or (at best) destroy Jabba as promised. He wouldn’t have told any of the others more than they needed to know.
Here’s how it might have gone down:
- Lando infiltrates Jabba’s Palace undercover. He can reconnoiter,
provide intelligence, that sort of thing. Note that he doesn’t break
cover until Luke springs his ambush.
- The droids deliver themselves. C3PO knows nothing of the plan besides
Luke’s message; R2D2 probably knows the whole thing.
- When being assigned, R2D2 deliberately mouths off to the chief droid,
probably hoping to get himself assigned to the sail barge. (The chief
droid treats it as a punishment berth.) Lando might even have
conveniently created an opening in the barge’s droid crew.
- Leia, disguised, brings in Chewie. She probably believes that she’s
supposed to free Han, then contact Lando. She and Lando would then
break Chewie out of his cell, grab the droids, and bolt. Luke had to know
that wouldn’t work.
- Luke shows up to demand their release, knowing that also was unlikely
to work. The rancor pit was (possibly) a surprise. (He did
deliberately stand on the hatch.) Either way, killing the rancor was
sufficient to torque off Jabba enough to put them all in one place.
- He gives R2 the cue to launch the lightsaber, and the "then we fight
them all and win!" part of the plan comes through.
Like I said, flimsy fan theory.