Marty McFly: That's right, Doc. November 12, 1955.
Doc: Unbelievable, that old Biff could have chosen that particular date. It could mean that that point in time inherently contains some sort of cosmic significance. Almost as if it were the temporal junction point for the entire space-time continuum. On the other hand, it could just be an amazing coincidence.
Not only did Biff from alternate 1985 (1985'-Biff) remember that date, but Biff from 2015 did too.
Before 1955-Biff had met 2015-Biff, it had already been a memorable day for Biff — but not in a good way. Until that day, Biff was the top dog at school. He had a gang following his leadership, he had his eyes set on Lorraine Baines and didn't know that neither in the original nor in the revised (after Back to the Future I) timeline she would fall for him.
Then George McFly — the school's nerdy wimp — stands up to him, knocks him out, and it all goes downhill for him from there. So 2015-Biff surely had a reason to remember that date.
Now 1985'-Biff was the first Biff that didn't experience that downfall, but still had very good reasons to remember that faithful night, because it had been an even weirder day for him.
He was still knocked out by the school nerd, George McFly, but besides that, he met a mysterious old man who gave him Gray's Sports Almanac. That was the beginning of his way to the top, to riches. Then that man also told him to watch out for a kid and a scientist coming for it.
2015-Biff: And don't tell anyone about it either. Oh, and there's one more thing. (Biffs leave the garage, conversation trails off) One day, a kid, or a crazy wild-eyed scientist who claims to be a scientist is gonna come around asking about that book...
So by all means, it is a memorable date, one that Biff surely would remember.