Tom Riddle, as a wizard child, was an orphan left in a muggle orphanage. The orphanage's administration was not aware of Riddle's condition.
The wizarding world should have been aware of this situation, because Dumbledore knew he had to visit the orphanage when the time was right to invite him to Hogwarts.
Since wizard children have a fairly good potential of uncontrollably showcasing magical skills, how could the Ministry of Magic, knowing this, leave a magical child under the care of unaware muggles, given the risk of violating the International Statute of Wizarding Secrecy?
Doesn't the Wizarding world have orphanages of its own? Or some kind of adoption program?
On a related note, I'm concerned about how no Ministry representative came to check on Riddle when he began using magic on the other orphans. This might suggest that the Ministry didn't know about Riddle, but then, how did Dumbledore know?