At what point and why does it become (almost) impossible to transport through shields in the Star Trek Universe?

Question

In all of the other series besides "Enterprise" the transporter only works with the shields down. I'm not going to quote the multitudes of episodes but it's a given to any Star Trek fan that shields have to be lowered in order to transport in or out.

In the series "Enterprise" however often times in the middle of battle the transporter is used with both ships shields raised.

For example in Enterprise "Divergence":

PHLOX: Admiral Krell. Doctor Phlox again. By now you may have noticed the

[Klingon Bridge]

PHLOX [OC]: Small package which we sent to your Bridge. What you may not know is that it contains a potent sample of the metagenic virus. Even as we speak it is dispersing throughout your ship, infecting you and your crew.

KRELL: You're lying!

PHLOX [OC]: Check your internal sensors.

This was transported onto the bridge of the Klingon battle cruiser in the middle of a battle. During the same battle Reed says, "It's no use. I can't penetrate their shields." In the series "Enterprise" I can give many many examples of people or things being transported on and off ships in the middle of battle with the shields raised. For example Archer transports with a crew onto the device built to destroy the Earth. In that same battle the Xindi Lizards transport Nosho off the Enterprise bridge in the middle of the battle. In the episode where the Orion Syndicate takes 9 Enterprise crewmen to sell as slaves, the Syndicate ships simply transports the crewmen off the Enterprise including T'Pol off the bridge with the shields raised in the middle of a battle.

Almost without question in the other series there had to be a reason given and it was an exception that such a thing could be done. For example in TNG if they had the shield frequency, or shield modulation, or some other explanation it could be done from time to time, but it was infrequent and almost always the shields being raised was used as an excuse that the transporter couldn't be used.

In "Enterprise" I often wonder why they didn't simply transport armed photon torpedos inside of other people's ships. Or for example why didn't the Enterprise simply transport a small bomb onto the bridge of the Klingon Battle Cruiser's bridge or next to its warp drive.

In the Xindi storyline, there were many times the Enterprise could have simply transported a torpedo or bomb onto an enemy ship or the device that could destroy the Earth. And visa versa the Xindi also had transporter technology and they could have done the same.

There's been some discussion below about whether the NX class has shields or not. I looked at Wikia and it listed the NX, Constitution, and Galaxy classes as all having deflector shields for defensive purposes. It does not list any other kind of shielding by name(other than hull plating also listed for the NX which I assume the others have incorporated in all the time). The NX clearly had deflector shields by the fourth season after the retrofit. Either way, almost every other ship in the "Enterprise" series had shielding, and I can put a bunch of quotes in to make this a very long question, but suffice it to say in almost every episode of "Enterprise" where there is a battle they are talking about penetrating an enemy's shielding, shield generators, etc.

There has also been some discussion about whether shields prevent a transport.

TNG "The Wounded":

RIKER: He'd never drop his shields and allow you to transport on board.

O'BRIEN: The Phoenix is using a high energy sensor system. It cycles every five point five minutes. Between cycles there's a window of a fiftieth of a second. Trust me, I can get through.

PICARD: Make it so.

[Phoenix Ready room]

MAXWELL: Not now! (O'Brien enters and Maxwell grabs a phaser)

O'BRIEN: I'm not armed.

MAXWELL: How the devil did you get over here?

So you see from this TNG episode that it's almost impossible, unless it's a transporter specialist, with the exact specifications of the other ships systems to transport through a ship's shields(there was no battle taking place and the Enterprise had its shields lowered at the time as well). Captain Maxwell was shocked since his shields were raised. This episode seemed a bit far fetched as it takes a full two seconds to complete the matter stream during a transport and a fiftieth of a second wouldn't give O'Brien enough time to get through, but that's a different question.

At what point and why does it become (almost) impossible to transport with the shields raised in the Star Trek Universe?

(If you feel like addressing my gleanings about why they didn't use this inability or ability however you view it, as a weapon more often, please feel free.)

Answer 1

You posit two questions, so I'll answer them separately:

Why

First of all, consider the technology of the NX-01:

The defensive systems of the NX-class were also not as advanced as those of other races at the time. Instead of deflector shields and tractor beams, the NX-class was equipped with polarized hull plating and grapplers.

(Source)

Then Memory Beta explains that:

While basic deflectors and deflector beams could provide minimal protection from weapons fire and basic mishaps, the majority of advanced starships were also fitted with more powerful shielding which could be activated in tactical situations. However, these shields were sufficiently powerful that they also blocked out transporter beams

So, even when later on the NX-01 does get deflector shields,it doesn't have the more powerful shields we see later on that could block these beams.

As to why it is possible to transport onto other ships and penetrate their shields, it is probably for a similar reason. Take Krell's ship, the D5:

The D5 class was protected by defensive shields and dispersive armor

This is a bit vague, but could be due to their not being particularly powerful relative to the 23rd century shields. Alternatively, it could be different shielding technology altogether; it is possible for the Borg and Dominion to transport through static Federation shields in the 24th Century (Source) so it's a possibility the transporters of the NX-01 were able to penetrate others' shields simply by the way they worked.

(Source)

When

Memory Alpha explains this a bit. As pointed out in the question, during the Enterprise era, transporting through shields was a possibility. Come the 23rd Century though, presumably due to shield advancements, it was not possible, that we have seen anyway, to transport whilst shields were up.

So, unfortunately the 'when' part of your question seems to be 'sometime after Enterprise and before The Original Series'

---

Interestingly enough though, it seems that this was reversed sometime around the Voyager era:

later starships such as the Intrepid-class and Sovereign-class vessels could transport personnel and objects freely to and from without having to lower their shields. (VOY: "Tsunkatse")

Although, as pointed out by Richard, this could simply be the ability to lower a small section of a shield.