According to the official novelisation, Han's response is driven by simple disbelief rather than denial. Note that the dialogue here is slightly different from the film.
“The Empire must have an outpost there,” Solo admitted. “Although,
according to the atlas, Alderaan had no moons.” He shrugged it off.
“Galactic topography was never one of my best subjects. I’m only
interested in worlds and moons with customers on them. But I think I
can get him before he gets there; he’s almost in range.”
They drew steadily nearer. Gradually craters and mountains on the moon
became visible. Yet there was something extremely odd about them. The
craters were far too regular in outline, the mountains far too
vertical, canyons and valleys impossibly straight and regularized.
Nothing as capricious as volcanic action had formed those features.
“That’s no moon,” Kenobi breathed softly. “That’s a space station.”
“But it’s too big to be a space station,” Solo objected. “The size of
it! It can’t be artificial—it can’t!”
For the record, Hans has been scanning the local area ever since their arrival so you'd expect him to have a reasonable idea of what's in his vicinity. In order for an object to be off the scope, but still visible to the naked eye it would have to be planetary in scale.
“It followed us!” Luke shouted.
“From Tatooine? It couldn’t have,” objected a disbelieving Solo. “Not in hyperspace.”
Kenobi was studying the configuration the tracking screen displayed. “You’re quite right, Han. It’s the short-range TIE fighter.”
