I'll lead with Dumbledore's description of the Quirrell incident:
[Harry]: "But why couldn't Quirrell touch me?"
[Dumbledore]: "Your mother died to save you. If there is one thing Voldemort cannot understand, it is love. He didn't realize that love as powerful as your mother's for you leaves its own mark. Not a scar, no visible sign ... to have been loved so deeply, even though the person who loved us is gone, will give us some protection forever. It is in your very skin. Quirrell, full of hatred, greed, and ambition, sharing his soul with Voldemort, could not touch you for this reason. It was agony to touch a person marked by something so good."
So I see two important points here. First, Harry didn't so much physically destroy Quirrell as cause him unbearable agony. And yes, I remember that this caused physical burns on Quirrell, but I don't think it would have destroyed him to the extent one would have to on order to destroy a horcrux. And second, this was so painful for Quirrell because he was sharing his soul with Voldemort. Horcruxes don't have souls (beyond the one stored in them), so they wouldn't be subject to the same pain, and being (for the most part) inanimate, they couldn't feel the pain anyway and so it couldn't destroy them. At most, the part of Voldemort's soul would "feel" the pain (if that were possible in some manner), but we know Voldemort couldn't even feel when they were destroyed entirely, so he probably wouldn't have felt that pain at all. And consider the part of Voldemort's soul that was inside Harry. If any of the horcruxes would have been destroyed by Harry's touch, would have been that one.
So in summary, I think it boils down to the fact that it was the pain that destroyed Quirrell, because of sharing his soul with Voldemort, and being inanimate objects with neither a soul to share nor the capability to feel pain, much less be destroyed by it, the remaining horcruxes (with the possible exception of Nagini) would not have been destroyed by Harry's touch.