I suspect that Tolkien was simply applying our universe's aversion for the number thirteen to his fictional universe.
However, The Letters of JRR Tolkien give a couple of examples of thirteen being less than auspicious. As they both occur earlier than The Hobbit, I think they can be considered in-universe explanations.
When Aulë secretly creates the first dwarves, he creates thirteen before Ilúvatar stops him.
Aulë, for instance, one of the Great, in a sense 'fell'; for he so desired to see the Children, that he became impatient and tried to anticipate the will of the Creator. Being the greatest of all craftsmen he tried to make children according to his imperfect knowledge of their kind. When he had made thirteen, God spoke to him in anger, but not without pity : for Aulë had done this thing not out of evil desire to have slaves and subjects of his own, but out of impatient love, desiring children to talk to and teach, sharing with them the praise of Ilúvatar and his great love of the materials of which the world is made.
The Letters of JRR Tolkien Letter 212
Ar-Pharazôn, the King of Númenor who rebelled and attacked Valinor resulting in the destruction of Númenor, was originally the thirteenth King.
Tar-Calion (the Quenya name for Ar-Pharazôn) was originally the thirteenth ruler of Numenor; in later developments of the history of Númenor he became the twenty-fifth.
The Letters of JRR Tolkien Note 131
I understand that these are somewhat tenuous connections. However, I suspect that if we had asked Tolkien why the number thirteen was considered unlucky in Middle-earth, he would have used similar logic to come up with an answer.