So all wizards get an invitation to Hogwarts at 11... How do they know whom to invite?
If a wizard is Muggle-born, then nobody would know about their existence.
Or do they register some weird magical stuff being done by them before 11?
As per JKR, there's a way (and it's McGonagall, not Dumbledore, who finds out for some reason)
... Also how does the Ministry of Magic find out these kids have powers?
... JKR. The Ministry of Magic doesn't find out which children are magic. In Hogwarts there's a magical quill which detects the birth of a magical child, and writes his or her name down in a large parchment book. Every year Professor McGonagall checks the book, and sends owls to the people who are turning 11.
(source: Online chat transcript, Scholastic.com, 3 February 2000)
Whenever a magical child is born, a magical quill writes that child's name down in a ledger of sorts. When the child turns eleven, he/she is sent a Hogwarts letter. There is no canon information regarding the other Wizarding schools, such as Beauxbatons or Durmstrang. Scholastic Chat 02.03.00
In addition to the two existing answers, there’s some Pottermore sample content which goes into more detail. It’s titled the Quill of Acceptance and the Book of Admittance (no account required). It starts as follows:
In a small locked tower, never visited by any student at Hogwarts, sits an ancient book that has not been touched by human hands since the four founders placed it there on completion of the castle. Beside the book, which is bound in peeling black dragon-hide, stands a small silver inkpot and from this protrudes a long, faded quill. These are the Quill of Acceptance and the Book of Admittance and they constitute the only process by which students are selected for Hogwarts School of Witchcraft and Wizardry.
Essentially, the Quill and the Book are somehow able to detect signs of magic in young children, and their names are recorded in the book, without human intervention:
At the precise moment that a child first exhibits signs of magic, the Quill, which is believed to have been taken from an Augurey, floats up out of its inkpot and attempts to inscribe the name of that child upon the pages of the Book.
The Pottermore content also explains that the mechanism behind the Quill and Book are entirely secret (to avoid arguments with disgruntled parents), and how this system ensures Squibs aren’t admitted by accident. I recommend reading it in full.
Then, as per the interview cited in the other two answers, the admissions professor (in Harry’s case, McGonagall) looks up children in the book every year:
In Hogwarts there's a magical quill which detects the birth of a magical child, and writes his or her name down in a large parchment book. Every year Professor McGonagall checks the book, and sends owls to the people who are turning 11.