TLDR: Yes, Valyrian steel is visibly different from even the best steel.
In the Books
In the books, Valyrian steel has a noticeable "ripple" effect to the metal, probably similar to a Damascus blade. It's also been described as having a "smoky" look.
Also, it was stated by a blacksmith that
the two longswords forged from Ice (Oathkeeper & Widow's Wail) would not hold the crimson color requested by Tywin Lannister, but instead kept reverting to a smoky merger of crimson and silver.
Thus we can assume that the metal has unique qualities causing this effect. Anyone who knows what to look for could therefore easily tell Valyrian steel from even the finest non-Valyrian steel.
On the TV Show
The Valyrian swords in the TV show appear to have a much higher polish to the blade. The higher polish causes it to be much more reflective, almost a chrome effect. In fact, in several scenes, Longclaw appears to almost have a glowing effect to it - very subtle, but there nonetheless.
This is most noticeable in Season 4 during the
Raid on Craster's Keep. Both outside, as well as when inside fighting Karl, the blade has an almost imperceptible white glow to it.
Also, as pointed out in the comments, when presented with it Jaime Lannister is able to recognize Oathkeeper as a Valyrian weapon by sight alone.
Removed video - no longer available