You're misreading Luke's sentiments in Ep 4. While Luke does want to follow his friends to the "Academy" (the only logical specific academy being one of the Imperial-run institutions such as the Academy of Carida or the Coruscanti Pilot Institute), it's not necessarily for the purpose of joining the Imperials. Luke, in the same movie, declares his hatred for the Empire. In the novel, there is a conversation between Biggs (who hasn't left yet) and Luke, where Biggs says he has a friend of a friend who has contacts with the Rebel Alliance at the Academy, and though Luke expresses misgivings about the hookup ("this twice-removed friend could be an Imperial agent. You'd end up on Kessel, or worse") this knowledge forms the impetus behind Luke wanting to leave Tatooine himself. And regardless of the machinations behind the scenes, Biggs does indeed show up on Yavin via the Academy.
On Dagobah, Yoda is trying (and not always succeeding) to get Luke to suppress emotion. While the training regimen Luke undergoes on Dagobah is not exactly what was sanctioned by the Jedi Academy in the Republic days, all the key elements are there; the physical training, meditation, learning about the perils of the Dark Side, struggling with those elements within yourself, etc. However, Luke does have some pretty well-ingrained lessons to "unlearn"; chief among them is that he usually refuses to abandon logic and emotion in favor of faith. Logic says a 10-ton X-wing is too heavy to lift out of a swamp without heavy equipment, so Luke didn't believe he could do it himself and couldn't believe it when Yoda did. Logic says when you're heading somewhere dangerous, you go prepared, so Luke didn't believe Yoda when he said Luke wouldn't need his weapons. On the flip side though, Luke wouldn't listen to logic saying he would be tempted by the Dark Side and was particularly susceptible at the time; he believed he could help his friends and so abandoned his training to go help them out of his (platonic) love for them.
Luke's black clothes in RotJ basically toe the line between appearing simple and humble and being seen as dangerous. Remember that Luke's very first in-person scene in RotJ has him using Force Choke (a "Dark" skill) against Jabba's Gamorrean guards. While not being flamboyantly obvious, Luke's black clothes make him stand out from just about anyone else in Jabba's court. However, he does keep the black jumpsuit for most of the rest of the movie (except while sneaking around Endor, where the Rebel camo was more appropriate), which is pretty heavy symbolism about the uncertainty of Luke's fate. He's weighted down with this concern over his father, he could still fall to the Dark Side himself (and is one swing of the saber away from it), etc.
Yoda wasn't surprised, per se, at Luke's affirmation of being a Jedi; he found Luke's statement humorous, probably because he'd heard similar self-inflating statements from his students many times over his life. Those students would have known that there were a series of trials to undergo, but consider Obi-Wan, who in his desire to help Qui-Gon bring Anakin into the Order volunteered for the trials, perhaps a bit overconfidently.
As far as Luke's intentions, we can only take them at face value; he wanted to confront his father to redeem him from the Emperor, and if he could take the Emperor out of the picture too, so much the better. Those actions were just as honorable as Anakin's, wanting to end the war, bring peace to the galaxy, protect the Republic from "enemies foreign and domestic". Anakin's desires were twisted by Palpatine until he'd accomplished the exact opposite. Luke's desires strayed dangerously along the same path, but Luke hadn't been under the Emperor's influence nearly as long as Anakin had, so all the Emperor could really do was to goad him into anger over how everything was going out beyond the observation window, to show Luke by his own actions how powerful he could become by using his hatred. He was as subtle as he could be, but nowhere near as insidious as he proved to be when corrupting Anakin. Even then, only one thing stopped Luke; realizing, by noticing Vader's severed mechanical hand, and then his own, just how much like Vader he'd already become.