In short, the Elves don't breed fast enough and their woodland is being decimated by the encroachment of human farms and settlements. 

They've been using advanced magic to arrest the decline of their civilisation, a decline that had been ongoing for millennia. They've even brought Middle-Earth to the brink of destruction by crafting magical rings in a vain attempt to block out the real world. 

Galadriel (in LotR) makes it abundantly clear what they expect to happen if the One Ring is destroyed and their rings are de-powered : 

>"Yet if you succeed, then our power is diminished, and Lothlórien will fade, and the tides of Time will sweep it away. We must depart into the West, or dwindle to a rustic folk of dell and cave, slowly to forget and to be forgotten."

Tolkien echoed this sentiment in a [letter][1] to a fan

>They [the elves] wanted to have their cake and eat it: to live in the mortal historical Middle-earth because they had become fond of it … and so tried to stop its change and history, stop its growth, keep it as a pleasaunce.”


  [1]: http://excerpts-from-tolkien.tumblr.com/post/33363275634/some-reviewers-have-called-the-whole-thing-the