The first time, it seems Hagrid was simply expelled, not charged with anything.
It seems like Hagrid was only expelled from Hogwarts for his suspected involvement in the attacks, not charged with anything by the Ministry of Magic.
“Hagrid wasn’t a fully qualified wizard; he had been expelled from Hogwarts in his third year, for a crime he had not committed. It had been Harry, Ron and Hermione who had cleared Hagrid’s name last year.”
- Harry Potter and the Prisoner of Azkaban, Chapter 5 (The Dementor)
Tom Riddle (while he's bragging about how he got away with opening the Chamber of Secrets) says "you can imagine how it looked to old Dippet", the Headmaster at the time. Neither Tom, Hagrid, nor anyone else mentioned any Ministry officials getting involved in the situation.
“It was my word against Hagrid’s, Harry. Well, you can imagine how it looked to old Armando Dippet. On the one hand, Tom Riddle, poor but brilliant, parentless but so brave, school Prefect, model student; on the other hand, big, blundering Hagrid, in trouble every other week, trying to raise werewolf cubs under his bed, sneaking off to the Forbidden Forest to wrestle trolls.”
- Harry Potter and the Chamber of Secrets, Chapter 17 (The Heir of Slytherin)
Even Rita Skeeter, who wanted to write as inflammatory a story as possible, doesn't mention Hagrid being charged with anything - which would certainly add to the scandalous nature.
“Rubeus Hagrid, who admits to being expelled from Hogwarts in his third year, has enjoyed the position of gamekeeper at the school ever since, a job secured for him by Dumbledore.”
- Harry Potter and the Goblet of Fire, Chapter 24 (Rita Skeeter's Scoop)
Hagrid did have his wand snapped and was banned from doing magic, but nowhere is it said that it had been a Ministry decision. It seems like his wand being snapped and him being banned were both related to him being expelled, and was likely a decision by the school, not the Ministry.
“Oh, well – I was at Hogwarts meself but I – er – got expelled, ter tell yeh the truth. In me third year. They snapped me wand in half an’ everything. But Dumbledore let me stay on as gamekeeper. Great man, Dumbledore.”
- Harry Potter and the Philosopher's Stone, Chapter 4 (The Keeper of the Keys)
Ollivander, as well, seems to associate his getting his wand snapped with his expulsion from Hogwarts.
“Good wand, that one. But I suppose they snapped it in half when you got expelled?’ said Mr Ollivander, suddenly stern.”
- Harry Potter and the Philosopher's Stone, Chapter 5 (Diagon Alley)
Further supporting this is that Dumbledore tells the Ministry at Harry’s trial for casting the Patronus Charm at Little Whinging that the Ministry isn’t responsible for “misdemeanors at school” and doesn’t have the power to expel students.
“The Ministry does not have the power to expel Hogwarts students, Cornelius, as I reminded you on the night of the second of August,’ said Dumbledore. ‘Nor does it have the right to confiscate wands until charges have been successfully proven; again, as I reminded you on the night of the second of August. In your admirable haste to ensure that the law is upheld, you appear, inadvertently I am sure, to have overlooked a few laws yourself.”
- Harry Potter and the Order of the Phoenix, Chapter 8 (The Hearing)
The decision to expel Hagrid would have therefore been with Hogwarts, not the Ministry.
The second time, he was taken straight to Azkaban - any trial would've been after.
The second time the Chamber of Secrets is opened, Hagrid is taken straight to Azkaban, without a trial before being taken there.
“I’m under a lot of pressure. Got to be seen to be doing something. If it turns out it wasn’t Hagrid, he’ll be back and no more said. But I’ve got to take him. Got to. Wouldn’t be doing my duty –’
‘Take me?’ said Hagrid, who was trembling. ‘Take me where?’
‘For a short stretch only,’ said Fudge, not meeting Hagrid’s eyes. ‘Not a punishment, Hagrid, more a precaution. If someone else is caught, you’ll be let out with a full apology …’
‘Not Azkaban?’ croaked Hagrid.”
- Harry Potter and the Chamber of Secrets, Chapter 14 (Cornelius Fudge)
He's told that he'll be let out "if someone else is caught", which could mean he'd have a trial, but could also mean that there was no trial planned for him, and he'd only be released if someone else was caught whether or not he could be proven innocent.
Then, when the basilisk was killed, Dumbledore wrote a letter to get Hagrid back, and Hagrid was returned to Hogwarts by three o'clock that same morning.
“What you need, Harry, is some food and sleep. I suggest you go down to the feast, while I write to Azkaban – we need our gamekeeper back.”
- Harry Potter and the Chamber of Secrets, Chapter 18 (Dobby's Reward)