Midi-chlorians do have an influence on how strong someone is in the Force, but even Anakin would never get to use the Force to a large extent if he did not work to develop it.
Everyone has some midi-chlorians, since that is similar to carbon in the Star Wars universe, where life cannot exist without it. As Emperor Palpatine said:
Darth Plagueis was a Dark Lord of the Sith, so powerful and so wise he could use the Force to influence the midi-chlorians to create life ...
Your midi-chlorian count is your potential with the Force, not your power. If you had a very small midi-chlorian count but worked super hard with the Force, you may be able to lift a small rock. But if you had a super high midi-chlorian count, then you can do much more powerful things as long as you work on it.
If you don't try, then you can't develop any powers even if you have a super high midi-chlorian count.