If a student came up to Dumbledore and said "I don't want to study anymore" would they be allowed to leave?
Can students drop out of Hogwarts?
Yes. At least up until HP and the Deathly Hallows, when going to Hogwarts is made compulsory by Voldemort's regime. As Lupin tells us:
“Attendance is now compulsory for every young witch and wizard,” he replied. “That was announced yesterday. It’s a change, because it was never obligatory before. Of course, nearly every witch and wizard in Britain has been educated at Hogwarts, but their parents had the right to teach them at home or send them abroad if they preferred.”
-- HP and the Deathly Hallows, Chapter 11: The Bribe (emphasis mine)
Of course, the decision is likely to be made by the parents rather than the student. While a child is still a minor, it is their parents' responsibility to ensure that they receive education. So a student who came up to Dumbledore and said they didn't want to attend Hogwarts any more would probably only be able to leave if they had their parents' support.
In fact, parents withdrawing their children from Hogwarts actually happened in canon:
"They probably want to look as though they're doing something," said Hermione, frowning. "People are terrified - you know the Patil twins' parents want them to go home? And Eloise Midgeon has already been withdrawn. Her father picked her up last night."
-- HP and the Half-Blood Prince, Chapter 11: Hermione's Helping Hand
And after Dumbledore's death:
All lessons were suspended, all examinations postponed. Some students were hurried away from Hogwarts by their parents over the next couple of days - the Patil twins were gone before breakfast on the morning following Dumbledore's death and Zacharias Smith was escorted from the castle by his haughty-looking father. Seamus Finnigan, on the other hand, refused point-blank to accompany his mother home; they had a shouting match in the Entrance Hall which was resolved when she agreed that he could remain behind for the funeral.
-- HP and the Half-Blood Prince, Chapter 30: The White Tomb
This was also confirmed by JK Rowling on her old website (emphasis mine):
Everyone who shows magical ability before their eleventh birthday will automatically gain a place at Hogwarts; there is no question of not being ‘magical enough’; you are either magical or you are not. There is no obligation to take up the place, however; a family might not want their child to attend Hogwarts.
Note that Muggle parents probably can't withdraw their children from Hogwarts, but this makes sense: they wouldn't be able to give their child magical training at home, and magical training of some sort, whether or not it's at Hogwarts, is necessary for a young wizard. Which brings me to ...
Can students drop out of studying?
No. Or at least I would assume not. In Britain, it's compulsory for children to have an education, but not for them to attend a school. The operative phrase, which was been in UK law ever since the Education Act of 1944, is "either by regular attendance at school or otherwise". The wording of this Act inspired the organisation Education Otherwise, a charity dedicated to aiding families whose children are educated "otherwise". As seen in the current Education Act of 1996:
The parent of every child of compulsory school age shall cause him to receive efficient full-time education suitable—
(a) to his age, ability and aptitude, and
(b) to any special educational needs he may have,
either by regular attendance at school or otherwise.
While this is of course a Muggle law, it's reasonable to suppose that a similar rule applies in the British wizarding world. Especially since a magical education is even more important than a Muggle education - if you don't learn how to use your magic properly, you could end up like Ariana, either dying or killing someone by accident.