Tolkien himself addresses this in an essay called "Laws and Customs Among the Eldar". There are essentially two problems:
- Elvish libido diminishes over time. Contrary to popular belief (and outward appearance, from a human perspective), Elves do age; and as they get older, they get less interested in procreation.
- Having children is exhausting. The Elves believe that their child-bearing process draws a heavier physical and spiritual toll on them, compared to what it does to humans. They're understandable reluctant to have a lot of kids, because what's the point in having a dozen kids if doing so might literally kill you?
From "Laws and Customs":
It might be thought that, since the Eldar do not (as Men deem) grow old in body, they may bring forth children at any time in the ages of their lives. But this is not so. For the Eldar do indeed grow older, even if slowly: the limit of their lives is the life of Arda, which though long beyond the reckoning of Men is not endless, and ages also. Moreover their body and spirit are not separated but coherent. As the weight of the years, with all their changes of desire and thought, gathers upon the spirit of the Eldar, so do the impulses and moods of their bodies change. [...]
Also the Eldar say that in the begetting [conceiving], and still more in the bearing of children, greater share and strength of their being, in mind and in body, goes forth than in the making of mortal children. For these reasons it came to pass that the Eldar brought forth few children; and also that their time of generation was in their youth or earlier life, unless strange and hard fates befell them.
History of Middle-earth X Morgoth's Ring Part 3: "The Later Quenta Silmarillion" Chapter 2: "The Second Phase" Laws and Customs Among the Eldar
That last point about the toll on their mind and body is worth expanding on, and fortunately Tolkien does this later in the essay:
[A]ll the Eldar, being aware of it in themselves, spoke of the passing of much strength, both of mind and of body, into their children, in bearing and begetting. Therefore they hold that the fëa [soul], though unbegotten, draws nourishment directly from the fëa of the mother while she bears and nourishes the hrondo [body; later changed to the word hroä], and mediately but equally from the father, whose fëa is bound in union with the mother's and supports it.
History of Middle-earth X Morgoth's Ring Part 3: "The Later Quenta Silmarillion" Chapter 2: "The Second Phase" Laws and Customs Among the Eldar