The elves were going to "the hill above Woodhall" and Woodhall is nearly due west of the Ferry. The road goes South from Hobbiton where it meets The East Road. The East Road goes nearly East and crosses the Brandywine Bridge.
See for instance: http://www.oocities.org/garfield1288/shire_map.jpg
(excerpt)

They run into the elves on or near the road that cuts through the trees in Woody End, somewhere close , but to the East of the branch. They took the "right branch to Woodhall" around "twilight."
They walked some, hid from the Black Rider, found the Elves, and walked some more to the Hill where the elves appeared to have a prepared place (a Grey-Haven Highway Authority Welcome Center)
The Hill was slightly North of the direct line to Buckland, but not really out of their way, considering the safety it afforded them.
EDIT:
Regarding the elves in question: they are not on their way to the Grey Havens. "We too are only tarrying here a while, ere we return over the Great Sea. But some of our kinsfolk still dwell in peace in Rivendell." They are living in the region and indications from the conversation are pretty clear that they were there even 17+ years prior when Bilbo left. At one point on the Hill in the evening, Gildor tells Frodo that he last said farewell to Bilbo on that very spot where they sat talking.
Gildor can easily be 1,000+ years old, so the exact duration of "a while" might mean something different to him than to you or me.