In Star Wars IV: A New Hope, after realizing that the Rebels were focused on a particular objective, Darth Vader, apparently without consulting his superior Grand Moff Tarkin - who incidentally kept pensive and silent - took two Wingmen and piloting an experimental modified early prototype of the starfighter TIE Advanced x1, diverted Tarkin's war plans and consequently made the Empire lose not only the Battle of Yavin but also the astronomically expensive Death Star. Did these actions get him tried and convicted in the Empire's court martial?
The comic Star Wars: Empire 14: The Savage Heart tells the story of what happened to Vader after being flung from the Death Star I and his return to the Imperial Military. As he arrives on Coruscant, he seems very much in charge and still in command. The story doesn't delve into any Imperial after-action reports like we see Imperial Cadet Pilot Han Solo go through in the deleted scene from Star Wars: Solo.
He was punished in other ways.
Firstly, I disagree with some of the points made in your question.
Darth Vader, apparently without consulting his superior Grand Moff Tarkin
While Tarkin might have held a higher position than Vader specifically on the Death Star, Vader's rank is nebulous enough that he didn't seem required to seek express permission from Tarkin to perform his normal duties.
took two Wingmen and piloting an experimental modified early prototype of the starfighter TIE Advanced x1,
diverted Tarkin's war plans and consequently made the Empire lose not only the Battle of Yavin but also the astronomically expensive Death Star.
Tarkin grossly underestimated the Rebellion's capabilities. Without Vader and the ships under his command, the Rebels likely would have had a much easier job.
I would argue various parties held more blame for the Death Star's destruction than Vader. Various Imperials championed the project without accounting for its flaws. Krennic, who oversaw the station's construction, failed to realize chief engineer Erso's treachery or the weakness he implemented in time. Tarken, as mentioned, failed to properly defend the base or authorize a full evacuation.
Vader, due to his high position and lack of specific rank, was not subject to the same bureaucracy as other Imperials. In From a Certain Point of View: The Empire Strikes Back, a couple of officers discuss how it would ultimately be futile to report him for executing Ozzel.
However, despite all of these factors, the Emperor did place blame on Vader, as he was one of the sole survivors of the incident. However, it was because of his plan to intentionally let the Falcon escape, not for the reasons you mentioned. The 2015 Darth Vader comic series explores the consequences of his failure in detail.
But you are the only one living to suffer my anger. You, an isolated survivor of the greatest military disaster in all my Empire's history? Oh, you are truly the Chosen One, Lord Vader. Chosen to be the one responsible - The Emperor, Darth Vader 1
His punishment was being put under the command of Cassio Tagge and being placed against various challengers to his position.