I was looking at the Noldor House of Finwë descendancy and on one side it shows Arwen, daughter of Elrond as we all know. But I didn't know this part: the other side breaks off into Aragorn as a descendant of Elros - Elrond's twin brother. Does that make them cousins?
The short answer is yes, they're directly related (first cousins) albeit separated by a very considerable age gap.
This handy family tree from The Encyclopaedia of Arda should be helpful to you:
Please note that the dotted line between Elros and Aragorn represents many thousands of years of lineage on Aragorn's side of the family.
Using the family-tree from http://lotrproject.com/, by my quick-count it seems Arwen and Aragorn II are first-cousins either 62 or 67 times removed, depending on whether you go through Aragorn's great x 14 grandfather (Arvedui) or great x 14 grandmother (Firiel) who both descended from Elendil (Arvedui on the Isildur side, Firiel on the Anarion side).
Hence Elendil is Aragorn's great x 38 grandfather as well as Aragorn's great x 43 grandfather.