I took Brand's statement to mean that the trip itself wasted decades of Earth time. Their Ranger had just been damaged by the wave and they lost one crew member, and on top of that, they realized that Dr. Miller had died shortly after landing (only a few hours before by the planet's time) and the planet was inhospitable. They should have been able to tell that not much time had passed on the planet (see herehere) since Miller had landed, and Cooper had suggested beforehand that they visit the other planets first because of the severe time dilation. So the trip was fruitless, and it took them 23 years.