If the Rebels knew how many people were on the Death Star, it would probably have been from the stolen Death Star plans. I haven't been able to find anything definitive, in part because there are so many stories about how the Rebels acquired the Death Star plans. We might get a more definitive answer once Rogue One comes out.
That said, the Rebels did acquire the Death Star plans with enough detail to reveal the battle station's thermal exhaust port weakness (which was only two meters wide). It would be rather odd for these plans to include such details without providing at least a rough estimate of the required crew for the Death Star (e.g. for manning all the defenses). The Rebels could probably also estimate the number of personnel from these plans even if an exact figure wasn't given in the plans.
Additionally, recall that R2-D2 hacked into the Death Star's computers in order to shut down the trash compactor that was about to crush Luke, Leia, Han, and Chewbacca. R2 may have acquired figures for the number of personnel while hacking the computer.
The Rebels probably didn't know the exact number of people on the Death Star, but between the Death Star plans and R2's hacking they probably had a rough estimate.