This is a bit like asking why the fundamental physical constants (gravitational constant, fine-structure etc.) have the values we measure. No physicist can really answer this. One approach is varieties of the antropic principle: if they didn't have those values (or similar ones), then human life wouldn't be possible, therefore we couldn't ask the question as to why they have those values.
An interesting variation to this has been proposed by Lee Smolin: he suggests that there might be an "evolution of universes", which in summary has it that (through somewhat complicated but not-too implausible means) such universes are preferred which happen to allow life to develop.
Now, in HHGTTG canon (right in the beginning of The Restaurant At The End Of The Universe), there is a somewhat similar idea:
There is a theory which states that if ever anyone discovers exactly what the universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable.
Later this is picked up, concluding that it means the Ultimate Question and Ultimate Answer simply can't be known at the same time in the same universe. And we might also read it thus that
- The pair of Question+Answer must be nonsensical / incompatible / wrongly calculated, in order to prevent removal of the discussed universe.
- Any universes in which 42 might not have been the answer would have been destroyed already in favour of ones where it is.
There is another theory which states that this has already happened.
Now, obviously the Odyssey of Arthur Dent's depends a lot on the bizarreness of that universe. More fundamentally, one might say the very existance of life needs certainly complexity. At any rate, we can say that Douglas Adams wouldn't have written about a less bizarre universe, implying that he would have chosen the one in which, by the aforementioned mechanisms, the Answer would need to be 42.