I don't think we know exactly how old Craster is, just that he's an old man well past his prime, so this might need some estimating based on how many generations of "wives" he has or what Mormont said about how long they've had this arrangement, etc.
I was reading How did the arrangement between the White Walkers and Craster begin? and it occurred to me that he's probably about the right age that his conception (by a rogue Night's Watchman with a Wildling north of the wall) might have co-incided with (or, been a direct or indirect consequence of...?) Brandon Rivers AKA Bloodraven's stint as a Night's Watchman, before whatever happened happened that left him stuck under a tree.
Not necessarily saying Bloodraven is the father: there are plenty of other possible links between Bloodraven going on the mission that ends with him stuck in a tree, the others / white walkers coming back, and a man being born who would then somehow end up with this relationship with them. That's beyond the scope here though.
Do these dates match up? Craster's description makes him sound to me maybe 20-30 years younger than Maester Aemon, which seems about right?
Also relevant: are there any other clues about who Craster's father might have been?