Aside from the other great answers, look at the political situation. Isildur was the king of the men in Middle Earth. The Elves had collected their friends of power to join them as allies and fight Sauron. After winning, they go off somewhere and rob - or maybe even kill- the king of a continent spanning nation "for the greater good".
Good luck avoiding constant war between men and elves after that!
