If At First by Peter F. Hamilton. I read it in The Solaris Book of New Science Fiction, which is where it was first published.
The story starts:
My name is David Lanson, and I was with the Metropolitan Police for twenty-seven years. When we got handed the Jenson case I was a Chief Detective, heading up my own team.
The time looper is Marcus Orthew. At the start of the story Lanson is put on a case investigating a break in at Orthew's office. The breakin was done by Toby Jenson who has worked out that Orthew can travel through time and is looking for evidence of it.
Jenson eventually convinced Lanson to apply for a search warrant to search Orthew's property, but by the time they get there Orthew is just disappearing into his time machine. Lanson tries to grab him and is swept up by the machine as well:
Marcus Orthew was standing inside the central chamber. Sort of. He was becoming translucent. I yelled at the others to get out, and ran for the chamber. I reached it as he faded from sight. Then I was inside. My memories started to unwind, playing back my life. Very fast. I only recognized tiny sections amid the blur of color and emotion— the high-speed chase that nearly killed me, the birth of my son, my dad’s funeral, the church where I got married, university. Then the playback started to slow, and I remembered that day when I was about eleven, in the park, when Kenny Mattox our local bully sat on my chest and made me eat the grass cuttings.
I spluttered as the soggy mass was pushed down past my teeth, crying out in shock and fear. Kenny laughed and stuffed some more grass in. I gagged and started to puke violently. Then he was scrambling off in disgust. I lay there for a while, getting my breath back and spitting out grass. I was eleven years old, and it was nineteen sixty-eight. It wasn’t the way I’d choose to arrive in the past, but in a few months Neil Armstrong would set foot on the Moon, then the Beatles would break up.
The music reference is when Lanson tells us:
So I fell back on the easiest thing in the world. I became a singer-songwriter. Songs are ridiculously easy to remember even if you can’t recall the exact lyrics. Remember my first big hit in 78, “Shiny Happy People?” I always was a big REM fan. You’ve never heard of them? Ah well, sometimes I wonder what the band members are doing this time around. “Pretty In Pink,” “Teenage Kicks,” “The Unforgettable Fire,” “Solsbury Hill”? They’re all the same; that fabulous oeuvre of mine isn’t quite as original as I make out. And I’m afraid Live Aid wasn’t actually the flash of inspiration I always said, either. But the music biz has given me a bloody good life. Every album I’ve released has been number one on both sides of the Atlantic. That brings in money. A lot of money. It also attracts girls. I mean, I never really believed the talk about backstage excess in the time I had before, but trust me here, the public never gets to hear the half of it. I thought it was the perfect cover.
And it ends:
See, I made exactly the same mistake as poor old Toby Jenson; I underestimated Marcus. I didn’t think it through. My music made ripples, big ripples. Everyone knows me, I’m famous right across the globe as a one-off supertalent. There’s only one other person in this time who knows those songs aren’t original—Marcus. He knew I came after him. And he hasn’t quite cracked the rejuvenation treatment yet. It’s time for him to move on, to make his fresh start again in another parallel universe. That’s why he framed me. Next time around he’s going to become our god. It’s not something he’s going to share with anyone else.
