Actually, it's incorrect to assume that Jay didn't remember his father at all. In fact, he distinctly has memories of his father from when he was a boy. As we find out later in the film,
Agent J's father, Major James Edwards Jr., was killed by Boris the
Animal during the launch of the Apollo 11 rocket. Young James Edwards
III (Jay) witnessed this, and the young version of Agent K then used a
neutralizer on him to erase the memory.
Given the nature of the situation, it's extremely like that either the MiB or the Air Force would have covered up the event. So, from the perspective of J's mother, her husband simply disappeared. She most likely assumed (incorrectly) that he left them. A young Agent J would have then grown up hearing how his father abandoned the family. Without any memories to tell him differently, J would consider that the truth.
You can see, then, how an adult J would resent his father. It also explains why the revelation of later events is so shocking to him. Not only does he discover
that his father did NOT abandon his family, he learns that the man was a hero who died to save Agent K from Boris the Animal.
THAT is the secret that Agent K wanted so badly to tell J but couldn't (as alluded to throughout the film) and why it was "an honor" to recruit and train Agent J.