The Baron Harkonnen mounted a huge planetary-wide invasion to take down the Atreides. Based on accounts of different characters, there had been more than 2000 ships and tens of battalions of Sardaukar involved. He later admits that it'll take at least 60 years of full-capacity spice production to offset the costs. However it's never explicitly mentioned how many men the Duke Atreides had in first place. Based on various scattered references in the preceding chapters, I guess this number has to lie somewhere between two and five thousand which seems too small to control a planet and certainly way too small compared to House Harkonnen.
As a side question, did he bring all his assets to Arrakis or were there troops and staff left on Caladan?