Sci-Fi Novel about time travel, an invisible house, and aliens trying to take over a distant human colonized planet

Question

I am trying to remember the name of a sci-fi book pertaining to a man whose planet was going to be attacked by aliens in a couple of years. To prevent his planet from being destroyed he was sent back to Earth (a far distance away) to get help. Upon his arrival on Earth he realized there was no help to be given. Earth was a planet full of academics and small people only focused on philosophy and things of the sort. The adventurous and large people were sent off decades ago to colonize other planets.

Later in the story he learns about time travel and cloning himself through travelling time. He brings the knowledge back with him to send the aliens 4 billion years in the past, where nothingness awaits them.

Explanation of Invisible House: One of these academic types invited him over to a house of science of some sort. This house was not actually invisible, but due to the time machine inside, the house was considered to be in a different "time". He had a special key to get in the house and once inside the main character and this academic introduced the possibility of being in multiple places / nowhere all at once.

Additions: I'm certain that this was one of my father's old sci-fi books. Other books I can remember being in his collection included Hyperion, I Robot, Pebble in the Sky, and Startide Rising. I don't know when it was published sadly and I can't figure for the life of me the authors name either.

Further Edit: It is not an Asiimov book or "The House That Stood Still", or "House of Suns"

Answer 1

This sounds a lot like Stepsons of Terra by Robert Silverberg.

I can't find much information on the net (a review here). From what I remember, it is about someone travelling to Earth from a distant colony to seek help. It also involves cloning via time travel.

...a man whose planet was going to be attacked by aliens in a couple of years. To prevent his planet from being destroyed he was sent back to Earth (a far distance away) to get help.

Already four worlds had fallen to the Klodni since the aliens had begun their campaign of conquest. The timetable said they would reach Corwin within the next decade.

...

Ewing shuddered as his tiny ship spiralled Earthward, bobbing in the thickening layers of atmosphere. Earth will save her colonies from conquest.

Earth was a planet full of academics and small people only focused on philosophy and things of the sort.

The more Ewing read, the less confident he became that he would find any aid for Corwin here. Earth had become a planet of gentle scholiasts, it seemed; was there anything here that could serve in the struggle against the advancing Klodni?

...the house was considered to be in a different "time". He had a special key to get in the house

Myreck held his key in front of him, stepped into the lot and groped briefly in mid-air, as if searching for an invisible keyhole. And indeed he seemed to find it; the key vanished for three-quarters of its length.

...

Myreck smiled. 'The house is three microseconds out of phase with the rest of the street. It always exists just a fraction of an instant in Absolute Past, not enough to cause serious temporal disturbance but enough to conceal it from our many enemies.

cloning himself

Ewing is kidnapped by Sirians (another colony). He is rescued by a disguised stranger. When he is shown the time machine he realizes that he was the rescuer and repeats the process.

He realized that he had several choices. He could walk out of the hotel room and leave Ewing-sub-two to his own devices, in which case, in the normal flow of events, Ewing-sub-two would awaken, be taken to Myreck's, request to see the time machine, and in due course travel back to this day to become Ewing-sub-one, rescuing a new Ewing-sub-two.

...send the aliens 4 billion years in the past, where nothingness awaits them.

He wondered briefly how the puzzled Klodni would react when they found themselves in the midst of a trackless void, without stars, without planets.

...

Ewing had hurled the Klodni fleet five billion years into the past. He shuddered at the thought, and turned his tiny ship homeward, to Corwin.