Likely, because he actually was. Remember the schtick of the episode is that the time passes for a cookie completely independent of what happens in "the real world", as we see with Matt torturing Greta by forcing her to live in 6 months of isolation over a period of about a minute. So, prior to the episodes' start, he jumps into the simulation, concocts a story to get them down to whatever they think that outpost is. They say, during the conclusion of the episode, that the 5 year period was only 70 minutes (a lampshade to the actual length of the episode).
As far as we know, Potter hadn't actually been inside Beth's father's cottage until the time of the crime itself - the four years prior to that, he had only been outside the cottage looking in.