It would be a matter for the Republic Supreme CourtSupreme Court to decide, and it's not clear what the decision would be. Both sides have a case, and it would come down to what each side could prove.
The Jedi attempted to kill Palpatine with no concrete evidence against him except that he was a Sith. The problem is that it was not a crime to simply be a Sith. Therefore the Jedi attempt to kill Palpatine was essentially an attempted assassination of the legitimately elected Supreme Chancellor of the Republic. The courts would probably define that as treasontreason. The exact text of Order 66 has not been given in canon, but if the Jedi were guilty of treason then the text of Order 66 as given in Legends certainly seems to apply:
In the event of Jedi officers acting against the interests of the Republic, and after receiving specific orders verified as coming directly from the Supreme Commander (Chancellor), GAR commanders will remove those officers by lethal force, and command of the GAR will revert to the Supreme Commander (Chancellor) until a new command structure is established.
Order 66Order 66, from the Legends novel Star Wars Republic Commando: True Colors
However, there are factors which make the case against the Jedi not as clear-cut as Palpatine would have the Supreme Court believe. In particular, Palpatine committed treason, too, as Darth Sidious in control of the Separatists (and he committed treason before the Jedi). If the Jedi could prove that Palpatine was fighting against the Republic as Darth Sidious then the court would have to decide whether or not this meant that Palpatine's claim to the office of Supreme Chancellor was legitimate at the time he gave the order. If the court decided against Palpatine on that point (which is likely) then Palpatine would have been ineligible to give the order.
Ultimately, the question is whether or not the Jedi would be able to prove that Palpatine was really Darth Sidious and that Darth Sidious led the Separatists against the Republic. If the Jedi could do that then the Jedi would win the case and Order 66 would be deemed as given illegally. If not, however, then the court would have to assume that Palpatine was the legitimate Supreme Chancellor and had the power and justification to issue the order. It's doubtful the Jedi would be able to prove their case since Palpatine/Sidious covered his tracks pretty well and (as Mace Windu stated) he had a lot of influence in the Senate and courts.
Using only the information available to the Republic characters in-universe, it's likely that Palpatine would prevail. Out-of-universe, though, the audience knows that Palpatine committed treason before he even became Supreme Chancellor and that therefore Palpatine did not have the authority to give the order.