The ability to destroy a planet is insignificant next to the power of the Force -- Darth Vader
Force users play an inordinate role in shaping the Galaxy whose history is defined by their actions: The Old Republic was 300 years after the Jedi Civil War and follows a "peace" of sorts due to the domination of the Jedi Order. The marginalization of the Sith was overturned by Palpatine with the destruction of the Jedi Order. The last Jedi, Luke Skywalker, later ended the Galactic Empire.
However, what are the factors which make Force users so special? Canon-wise, Force users appear powerful but not inordinately. For instance, during Order 66 groups of 7 or so droids could down a Jedi -- given how rare Jedi are as confirmed by this answer it would appear that the Jedi Order's total strength could be completely replicated (cheaply) with more droids or clones
Furthermore, highly trained warriors can defeat Force users in a one-on-one fight as is the case with Jango Fett (pictured below).
Luke's impressive feat of destroying the Death Star was also somewhat replicated by Han Solo and Wedge Antilles, implying that while the Force confers a valuable edge, it does not appear to be an insurmountable one.