It never made any sense that Trek shields weren't capable of blocking kinetic attacks.
Starship shields work very similarly to internal forcefields (which are more than capable of preventing any and all kinds of attacks).
We've seen Worf rig up a personal shield in TNG from a commbadge that managed to deflect several bullets.
Shuttles in comparison to a much larger starship would have a similar effect... they would run up against the shields and would effectively ram INTO the shields without penetrating them.
With the case of DS9... we do have at least 1 example of Jem'Hadaar kamikaze runs not being effective.
The Dominion's first attack (with the Cardassians) on DS9.
There was a scene where a Jem'Hadaar attack ship attempted to ram the station... and we've seen it clearly impact the station's shields without penetrating them (the station was at the time under heavy assault from the Dominion/Cardassian fleet no less).
In the episode where a Jem'Hadaar attack ship rammed a Galaxy class - by that point the crew already turned the shields off as they were useless against polaron beams and shuffled the energy into reinforcing Phasers instead.
That's why the attack ship was able to successfully take out the Odyssey.
As for subsequent scenes where we've seen Dominion kamikaze runs being effective... well, most of those scenes already occurred with both sides being under heavy fire and shields being mostly or completely drained.
DS9 VFX department was almost criminally negligent in showing shields during fleet battles - which is one of the reasons why single weapon shots would start taking out ships or make large holes in them (which simply wouldn't have happened that fast - but I guess that for the sake of the story, we've seen an abridged version of the battle where you get the general idea - although, with fleets, I think what the VFX team decided to do instead of showing shields glowing, they ended up just dissipating the shots into the hull without them doing anything).
Star Trek Nemesis however somehow managed to completely ignore a layered shield system on the Scimitar which allowed the Enterprise to get through as if they weren't there.
Though, that movie also showcased skin-tight shields, which might explain why the shield wasn't immediately visible on other sections of the Scimitar (because energy weapons seem to be dissipated throughout the shield, while a starship kamikaze run produces a more localized effect).
Still doesn't make sense to be honest, because the Scimitar's shields should have been A LOT more effective - but we don't know what Picard might have done off-screen to negate Scimitar's shields.
As for 'Year of Hell'... those were micro-meteroids... so navigational shields are used for moving space debris (matter) that travels at very high speeds out of the way of the ship primarily (although, they seem to simply REPEL and alter the trajectory of the said objects as opposed to allowing them to make contact in the first place), but they do seem to be effective against energy based attacks too (TNG mentioned that lasers aren't enough to even penetrate navigational shields)... just nowhere near on the level of actual defensive shields.
As for Beyond movie swarm... the crew did act surprised the shields were not effective, indicating that they should have been.
Its one of the things I don't like about these new movies... shields for example are portrayed as non-existent actually. If you look at the battle between the Enterprise and Vengeance for instance (Star Trek Darkness), you'd see that each phaser shot and torpedo makes contact on the hull itself as if the shields aren't there (and Kirk did order shields up with enough time just before the Vengeance started attacking).
Now, this could be explained in another way: the Vengeance was a top-secret Federation ship, and therefore could have had Enterprise shield frequencies, making them useless against their weapons.
But this doesn't seem likely... just as unlikely as the premise that detonating several torpedoes INSIDE the Vengeance wouldn't destroy it.
Ok... starships are highly advanced and made from really tough materials, so even these torpedoes could easily destroy a planetary surface in a few shots, on starships (even interiors) they wouldn't necessarily incur too much damage... however, 1 torpedo was detonated in close proximity to 71 others.
Durability aside, that SHOULD have destroyed the Vengeance right then and there, or at least vaporize an entire secondary hull, breach all its way to the Warp core and take out the entire ship (unless the ship had advanced internal forcefields capable of limiting such damage, but generally, I would imagine that the Vengeance's warp core detonating from inside it would be more than enough to overload any such defenses).
So, we have expamples of Trek shields being very effective against kinetic and kamikaze attacks (not just energy based weapons).
Problem is that in the case of the movies, things like this get overlooked for the sake of increasing the drama.
In 'Beyond' though, Krall already had information about Yorktown and the Enterprise, meaning he probably had what he needed to increase the swarm's effectiveness in negating the Enterprise shields (likely by modifying the swarm fleet with adaptive shield frequencies that would respond to several different Federation shield frequencies) - although its still a bit dubious as to how he was able to penetrate Federation comm frequencies and acquire sensitive data.
Advanced tech will take you only so far, and the Federation by the mid 23rd century already had it's technology from a mixture of dozens of different alien species.
I find it unlikely that such versatility and combination could be easily penetrated by Krall even if he did have underlying understanding of Starfleet - but as I said, things changed radically.
