Elrond's disdain for men is introduced in the films; it is not present at all in the books, nor would it make any sense. He only says that Isildur took the Ring 

> as should not have been

and that he 

> would not listen to our [Elrond and Cirdan's] counsel

(The Council of Elrond).  

Later in the same chapter, Elrond admits that he himself fears to take the Ring due to its malign influence. Moreover, at the time of the events of LotR, Elrond is over 6,000 years old, and is well aware that the elves have a history of creating (or aiding the creation of) rather dangerous magic items. The idea that he would blame men (or specifically Isildur) for this is absurd. After the last alliance, Elrond continued to work alongside men, in the fight against the Witch-King (see LotR appendix A parts (iii) and (iv)), and by sheltering the heirs of Isildur in Rivendell after the fall of Arthedain. During the Council of Elrond, he credits the Lords of Minas Tirith for their efforts in holding back the forces of Sauron.

On the other hand, in the films, Elrond does seem to distrust men, and this stems from Isildur's failure to destroy the ring. Referring to this event (the relevant scene starts at 1:26:22 in the extended edition of the Fellowship of the Ring), Elrond says

> I was there the day the strength of men failed.  

Later in the same scene, he makes a curious remark about Aragorn: 

> He turned from that path [reclaiming the throne] long ago. He has chosen exile. 

What this means is unclear; certainly it doesn't refer to anything in the books. We do know that Thranduil sends Legolas to seek him out at the end of the Hobbit trilogy. It's possible that in the film universe, Aragorn does something that earns him Legolas' respect, but we have no way to know what this is.