This is definitely Robert Heinlein's Space Cadet.
As part of the admissions process to join the Space Patrol, Matt Dodson has to take several tests.
One involves dropping beans into a jar while keeping his eyes closed.
Another is the one mentioned in the question.
Matt sits down at a table with a game made of a bunch of switches, lights, gates, and instructions. Rather than just jumping in and doing things, he tries to make a simple list of how to score in the game.
He writes down what amounts to a truth table, and realizes that there's no way to score any points at the game. He takes his notes and goes to ask the test proctor if maybe the wrong instructions were supplied. The proctor checks the time and tells Matt that his score has been noted and that he should go to the next scheduled test.
Some of the fellows talk about it later and realize that the tests often weren't testing what you thought they were.
The "impossible" test was testing your ability to reason and how methodical you were.
The "bean test" wasn't testing dexterity but rather honesty - the cadets were under observation though they didn't know it. It was easy to cheat and some thought they'd get away with it because no one was looking. The Patrol wants people who do what they are supposed to even when no one is there to see it.
Most of the tests were that way. They gave you some test, and observed how you went about it rather than what the end result was.
Space Cadet is one of my favorite books.
Besides the psychological parts, it also "looks into the future."
Matt and his friends all have and use cellphones (though they are just called phones.) Matt's mother even calls him at a bad time and embarrasses him in front of his new acquaintances on admissions day.
The space travel parts are somewhat off. It was written in 1948, before anyone had ever been off planet. It gets things wrong - Venus is hot but liveable, Mars is cold but has a native population and a breathable atmosphere, etc.
Still, it has a good story and gives you things to think about.
1948 was at the very beginning of the Cold War, when people were beginning to wonder what would come of nuclear weapons.
In Space Cadet, the solution was a sort of "United Nations" with teeth. The Patrol provides the "bite."
There are automatic atomic bombs in orbit around the Earth. Any nation that attacked another country would be bombed on orders of the "United Nations" type organization. The Patrol maintains the bombs and works to prevent things from reaching the point of national attacks and nuclear retaliation.
The motto of the Patrol is Quis custodiet ipsos custodes? - "Who watches the watchers?"
They are very aware of the power they hold and how easily it could be abused.
Space Cadet makes several references to the Heinlein short story The Long Watch in which a Patrol officer by the name of Dahlquist prevents members of the Patrol from putting the Earth under a military dictatorship headed by the Patrol. Dahlquist takes many of the atomic bombs that would have been needed out of commission to prevent their use in the takeover. In the process, he is exposed to a very large dose of radiation. Despite this, he holds his post until reenforcements arrive to capture the plotters.
Dahlquist is one of "The Three" - three names that are always called as part of every roll call in the Patrol. Each of "The Three" in some way represented the ideals of the Patrol. Dahlquist died of radiation poisoning to prevent a coup. Rivera went to negotiate in the capitol of a country threatening to attack another country in the hopes of preventing a war or retaliatory attack - he died when the Patrol nuked the capitol to stop the war. Wheeler was the third of "The Three." I can't remember what he did.
For a "young adult" story, "Space Cadet" takes on a lot of really hefty themes - and manages to be fun to read at the same time.