Borg are looking to achieve their idea of perfection: a melding of the ultimate technology and biology. To achieve this they must assimilate species with a high technological or biological quotient (or preferably both: see species 8472).
Because 99% of the technologically advanced species in our galaxy are physically very similar to the Borg, the Borg wait for them to develop significantly interesting technology. Coincidentally, most species that develop something interesting to the Borg have also developed and become dependent upon energy weapons as their primary means of attack and defense. In the process they have abandoned martial prowess and physical firearms as by-products of a less civilized age.
Because of this, almost any physical attack that a Borg is likely to be on the receiving end of will be ineffectual against their cybernetic bodies. This leaves energy attacks as the only conceivable threat, for which the drones have adaptive shielding.
There are obvious exceptions, of course. But as others have pointed out, effective physical attacks against the Borg are singular events caused by hero characters. Any time that a regular schmoe has tried to assault a Borg drone physically they have typically either been killed or assimilated.