To shortly answer the question - objects protected by the Fidelius Charm are most probably vulnerable to "collateral damage" coming from the surroundings. However the Death Eaters either had no conclusive information or were restraining themselves from drastic actions even after taking over power.
More detailed - we see three times where the Fidelius charm is applied:
- The Potters house - Voldemort already knew the secret so no need to apply mass destruction.
- Grimmauld number 12 when Harry, Hermione and Ron were hiding there - Voldemort had no conclusive information about that. All he knew was that the house was deserted. He still left some guards to keep it under watch but they never noticed anything worth reporting (remember that he provides very thorough punishment for false alarms).
- The Shell Cottage and The Burrow - as Neville mentioned - the Death Eaters try to spill as little pure blood as possible. They are rounding up muggle-borns because they have at least partial support from the community. An open attack on some pure blood family would have been a very unpopular move and there would be a risk of an open rebellion.
To answer the sub-question - does the Fidelius Charm provide physical protection from collateral damage - here some analysis:
Taking the description from Pottermore:
The Fidelius Charm is extremely ancient and still used to this day. It
involves the concealment of information inside a living person. The
chosen person, or Secret Keeper, is the only person who is thenceforth
capable of revealing the protected information to others, however many
previously knew it.
The main point of the Fidelius Charm is "concealment of information". No physical protection is mentioned.
Empirical evidence shows a bit different picture - there IS some kind of physical protection. The Death Eaters surely knew the location of Grimmauld number 12 as the house of the Black's family. They also knew that Sirius is the legal owner of the place as the last living descendant of the family. Nevertheless they were not able to access the place.
With all that I would call the Fidelius Charm "magical encryption".
In our world encryption includes having a secret key (consider it "the secret keeper") and applying some mathematics (known to the most people as "magic" :) ) so that certain information is converted into something completely unreadable. The one owning the secret key can easily read the information. For others it is EXTREMELY hard although not impossible to do so. In this case "extremely" means like 5-10-20+ years to break the encrypted data without owning the appropriate secret key. Encrypted data is not protected from destruction - everyone can freely damage or completely delete it so that it can no longer be recovered even via the secret key.
- "secret key" + "owner" -> "secret keeper"
- "encrypted data" -> "hidden object"
- "mathematics" -> "magic"
and you have the Fidelius Charm.
All in all - the Fidelius Charm is not a mean to withstand a siege. If cornered you would not be protected from indirect attacks. As an example - looks like the charm allows air to flow in and our of the protected building. Sirius lived for almost an year in the house and did not suffocate. So setting the surroundings of the protected building on fire would cause smoke to enter it and also eventually set it on fire. Using some mass destruction weapons will also destroy anything protected by the Fidelius Charm.