I am trying to remember the name of a sci-fi book pertaining to a man whose planet was going to be attacked by aliens in a couple of years. To prevent his planet from being destroyed he was sent back to Earth (a far distance away) to get help. Upon his arrival on Earth he realized there was no help to be given. Earth was a planet full of academics and small people only focused on philosophy and things of the sort. The adventurous and large people were sent off decades ago to colonize other planets.
Later in the story he learns about time travel and cloning himself through travelling time. He brings the knowledge back with him to send the aliens 4 billion years in the past, where nothingness awaits them.
Explanation of Invisible House: One of these academic types invited him over to a house of science of some sort. This house was not actually invisible, but due to the time machine inside, the house was considered to be in a different "time". He had a special key to get in the house and once inside the main character and this academic introduced the possibility of being in multiple places / nowhere all at once.
Additions: I'm certain that this was one of my father's old sci-fi books. Other books I can remember being in his collection included Hyperion, I Robot, Pebble in the Sky, and Startide Rising. I don't know when it was published sadly and I can't figure for the life of me the authors name either.
Further Edit: It is not an Asiimov book or "The House That Stood Still", or "House of Suns"