Your assumption on the letter sender is were your misunderstanding is occurring. 
In the Philosophers Stone, we find out Hagrid is in charge of sending Harry his letters. His first attempt added a letter addressed to Harry potter blahblah, under the staircase, to the pile of muggle mail, whether sent through mugglemail or simply slipped into the mailbox/slot is another question. 

So we have

 A. letters were sent directly to Harry,

B. the amount of letters increased in an attempt to get one of them into Harrys hands passed the Dursleys without direct intervention (Also Hagrid was allowed to use magic on this mission and he was most likely having fun (since he is technically not allowed to do magic, strictly speaking of course) ). 

C. Finally Hagrid does show up and personally deliver the final letter to Harry after the Dursleys made a run for it and were hiding on an abandoned island.