The Baron Harkonnen mounted a huge, planet-wide invasion to take down the Atreides. Based on accounts of different characters, there had been more than 2000 ships and tens of battalions of Sardaukar involved. He later admits that it'll take at least 60 years of full-capacity spice production to offset the costs. However, it's never explicitly mentioned how many men the Duke Atreides had in the first place. Based on various scattered references in the preceding chapters, I guess this number has to lie somewhere between two and five thousand which seems too small to control a planet and certainly way too small compared to House Harkonnen.
More specific to your initial question on the troop numbers there are no definite tables but 300,000 seems reasonable for a planet of no more than a few million.
There is a large expense to sending men across space and battles often devolve to hand to hand combat, this is also in the future where most individual powers have parity as far as technology. Meaning everyone can be pretty confident in the performance of their forces and troop numbers.
An army of 300,000 men dug in would probably be more than enough troops for any opponent. With the exception of an opponent + imperial sardaukar.