For this we need to look at the following things -
- Is Harry's class size indicative of average number of annual wizardly births?
- How large is the population of Wizarding UK anyway?
- Why does Diagon Alley always seem to be full?
- What's with the number of people in the Ministry?
Is Harry's class size indicative of average number of annual wizardly births?
This seems unlikely. Since Harry was born smack in the middle of one of the worst conflicts Wizarding UK faced. We know that people were disappearing every week for around 11 years, for a low estimate of around a 1000 adults people dead - which would be at least 10% (but more likely a lot more) of the population. Not only people too dead to spawn kidlets, but also not exactly an ideal time to be deciding to have babies (or be pregnant) anyway.
How large is the population of Wizarding UK anyway?
We have reason to believe average wizarding lifespan to be around 150 years; and assuming a conservative average annual birthrate over those 150 years to be around 50 gives us a population size of 7500. Which is not a huge number, but not exactly a tiny one either. Also consistent with comments by JKR.
Why does Diagon Alley always seem to be full?
Diagon Alley seems to be a street market style affair - and it definitely doesn't seem to be very large from what is described in the books. It wouldn't really take a very large number of people for such a place to appear crowded. But in addition to that, there's also the fact that Harry (and hence the readers) tend to see DA during what would possibly be one of the busier portions of the year for the market - the back-to-school shopping season. All those kids buying school supplies has got to create a bit of a crush.
What's with the number of people in the Ministry?
There seems a very large number of people in the Ministry compared to the total population, that's fairly true. But our window into the Ministry is through the Weasleys and the OoTP. Given that Arthur is a prominent civil servant and the OotP by its very nature would be made up of people likely to be Aurors or involved in politics, that seems at least one easy way to answer why so many of the adults we see, have something or other to do with the Ministry.
But in addition to that, there's also the question of the makeup of wizarding society. It's not a completely isolated population - it's a secret population within a much larger one. Most of the functions of the MoM seem to do with maintaining secrecy - so that's an extra function of the government that soaks up numbers - most of the Department of Magical Law Enforcement and Department of Magical Accidents and Catastrophes. Then there's also the Department for the Regulation and Control of Magical Creatures - given that these people need to deal and liase with whole societies of sapient magical people in addition to regulate dragons and other non-sapient magical creatures of that sort - their numbers being more significant than their Muggle counterparts doesn't seem too far fetched either.
Edited to add some numbers on Hogwarts and Wizarding Britain numbers from Jo: I found this interview from JKR regarding Hogwarts numbers:
Then I’ve been asked a few times how many people and because numbers are not my strong point, one part of my brain knew 40 [original characters she had created for Harry's year], and another part of my brain said, “Oh, about 600 sounds right.”
600 in Hogwarts doesn't seem so small after all. But that paints Harry's year as being more than half the usual yearly intake ((600-40)/6) of around 90, which then goes on to give up a total population size of 13,500, take around a couple of thousand for the casualties of the two Wizarding Wars in the last century, and that's still a fairly decent number.
But then that same interview goes on to say (in the very next line):
Let’s say three thousand [wizards/witches in Britain], actually...
What?! That means a Hogwarts year should have no more than 20 students at most! Harry's year is overpopulated; and total Hogwarts population should be something like a 150; and casualties under Grindelwald and Voldemort were massive percentage chunk of population. Or maybe most Wizards only live up to a 70-ish, same as Muggles, except a lucky few. That's the only way to explain those numbers.
Thankfully, JKR gives us an out and ask that we: "don’t hold me to these figures, because that’s not how I think."
In Conclusion: A 'corrected' average Hogwarts size of around 400-450 might be a good compromise between JKR and McNumbers. Harry's year in pretty small with 40 students, but the average yearly intake is 60-65 (two dorms per year per house per gender with 3-4 kids each (explains MWPP), instead of one with 5); leading to a total British Wizarding population of around 10000. The rest of the points in my answer still apply, I think.
And the explanation for this, as given by JKR herself:
Oh, Emerson, my maths is so bad.