There is no last Elf in Middle-earth - there's still Elves here!
Elves cannot live in Middle-earth indefinitely, embodied. Though the Three Rings helped keep them around and embodied, all Elves in Middle-earth were subject to fading, the loss of their corporeal bodies.
On earth the Quendi suffered no sickness, and the health of their bodies was supported by the might of the longeval fear. But their bodies, being of the stuff of Arda, were nonetheless not so enduring as their spirits; for the longevity of the Quendi was derived primarily from their fear, whose nature or ‘doom’ was to abide in Arda until its end. Therefore, after the vitality of the hröa was expended in achieving full growth, it began to weaken or grow weary. Very slowly indeed, but to all the Quendi perceptibly. For a while it would be fortified and maintained by its indwelling fëa, and then its vitality would begin to ebb, and its desire for physical life and joy in it would pass ever more swiftly away. Then an Elf would begin (as they say now, for these things did not fully appear in the Elder Days) to ‘fade’, until the fëa as it were consumed the hröa until it remained only in the love and memory of the spirit that had inhabited it.
But in Aman, since its blessing descended upon the hröar of the Eldar, as upon all other bodies, the hröar aged only apace with the fear, and the Eldar that remained in the Blessed Realm endured in full maturity and in undimmed power of body and spirit conjoined for ages beyond our mortal comprehension.
(History of Middle-earth, Volume X, Myths Transformed)
When an Elf faded, they had the choice of going West. But not all heeded the call.
But it would seem that in these after-days more and more of the Elves, be they of the Eldalië in origin or be they of other kinds, who linger in Middle-earth now refuse the summons of Mandos, and wander houseless in the world, unwilling to leave it and unable to inhabit it, haunting trees or springs or hidden places that once they knew. Not all of these are kindly or unstained by the Shadow. Indeed the refusal of the summons is in itself a sign of taint.
It is therefore a foolish and perilous thing, besides being a wrong deed forbidden justly by the appointed Rulers of Arda, if the Living seek to commune with the Unbodied, though the houseless may desire it, especially the most unworthy among them. For the Unbodied, wandering in the world, are those who at the least have refused the door of life and remain in regret and self-pity. Some are filled with bitterness, grievance, and envy. Some were enslaved by the Dark Lord and do his work still, though he himself is gone.
(HoME, Volume X, Laws and Customs of the Eldar)
So, the spirits of lingering and rebellious Elves are still haunting Middle-earth, indefinitely.
As to characters we know, probably the last Elf we know to have left (not the last elf to have been there), is either Círdan or Celeborn. It seems to have been Círdan's task to man the Last Ship, to ensure that every Elf that wanted to physically leave could do so, but at the same time in the Lord of the Rings prologue, we have this:
It is probable that Meriadoc obtained assistance and information from Rivendell, which he visited more than once. There, though Elrond had departed, his sons long remained, together with some of the High-elven folk. It is said that Celeborn went to dwell there after the departure of Galadriel; but there is no record of the day when at last he sought the Grey Havens, and with him went the last living memory of the Elder Days in Middle-earth.
Since Círdan was also around throughout the Elder Days, the implication is that either he and Celeborn left together, or Celeborn left after Círdan. It is also conceivable that, since the departure of Círdan and Celeborn is unrecorded, that they left after Merry and Pippin's death (sometime after 1484) but before Legolas left in 1541 - if we don't consider Legolas to have memories of the Elder Days. Notably, Legolas did build his own ship.