I always found it odd an Avada Kedavra curse, even rebounded, would damage the house. It wasn't supposed to leave any marks other than the death of it's victim, and this was part of it's terror. Additionally, descriptions that the curse itself damaged the house can be mainly explained as inferences made by the characters or Riddle messing with Harry's head (see the comments), meaning this does not have to be the accurate in-world explanation.
Therefore, I had always myself inferred that Peter was there with Voldemort — likely waiting outside, and possibly at Voldemort's instructions in order to be sure he had the right place — and it was actually Peter who destroyed the house.
There are several things going on that I feel support this idea.
First of all, a Peter who is waiting outside would not be immediately clear about what happened; he would need to investigate. Inside he would find a wand, a baby, and two dead bodies. I expect what would most grab his attention was the wand of his master, which he would see as an item of great power. Peter would try to use this wand, and I doubt the wand would accept him. At this point, the destruction of the house is a natural result.
Alternatively, he may have simply felt the need to destroy evidence — may have been under orders to do so, along with casting a Dark Mark, once Voldemort left. He may also have wanted to kill the baby himself, but being a coward he felt that a general destructive curse to the house would do that job rather than needing to more directly murder the child.
And then Hagrid arrives, followed a few minutes later by Sirius. Of course Peter has ability to hide from Hagrid. Once Hagrid leaves, this places Peter and Sirius together at the scene with no witnesses. Sirius in dog form would have no trouble sniffing out Peter's rat, and what happens next is now well understood.
The point of all of this is Peter was likely there first, with ample time to recover the wand. While it's not clear, there are some indications that an animagus could keep items like this on their person while transformed, making Peter's escape with the wand from the home and Sirius (with the appropriate distraction) easy enough to accomplish.
Even if this does not satisfy you as the most likely option, it should at least show there is room for interpretation here.