Originally, Starfleet ship brigs did not employ a force field, they were merely
small cells with solid doors (ENT episode: "Divergence")
By the 23rd century though, we know from TOS that force fields were implemented.
Memory Alpha describes the 24th century Starfleet starship brigs as being:
commonly covered on three sides by bulkheads with an
electrostatic force field on the fourth side, which could be turned
off and on as an entrance or exit way.
Considering what we see in TOS, I think it's reasonable to assume that the 23rd century starships had a similar composition - they also had a force field for entrance and seemed to have bulkheads.
Now, we only have the two elements in the brig - a force-field and the bulkheads. Considering the entire ship has bulkheads all over the place and they don't impinge on the ability for a transporter to work, we can safely assume it is something to do with the operation of a force field. This is supported, as pointed out in the question, in the TNG episode The Hunted where:
Worf ... calibrated the transporter beam to activate 0.1 seconds after
the cell's force field ... shut down.
The Memory Alpha page on Force-Fields doesn't add much, stating what we already know:
If a force field is active, an object or transporter beam generally
cannot pass through it, although this is not always the case. The Borg
are known to not be hindered by most force fields; they can adapt and
simply walk through the field. However, a Borg drone can be cut off
from the Borg Collective if it is surrounded by a Starfleet level 10
force field. (DS9: "The Jem'Hadar"; TNG: "I Borg")
An important point to take form this quote though is that not all force-fields block transporter beams.
Another important point from this page is that
If necessary, a transporter platform can be completely enclosed within
a force field, to, for example, contain an alien subject or prisoner
during transport. (TNG: "The Hunted", "Realm of Fear"; VOY: "Death
So a transporter beam can still operate despite a force field if it is on the transporter pad.
So we have some slight inconsistencies here. The important point though is, how does a force-field actually work?
I think it's pretty clear that a force-field works by preventing matter from entering/exiting a certain area. We also know that there are different levels of force fields, being
rated by intensity, ranging in strength from level 1-10. A level 10
force field is the strongest and would be used, for example, during a
scientific experiment of which the outcome was unknown, or known to be
explosive in nature
Now, I interpret 'strongest' as referring to strength, but also the level of particles which pass through the field. Bear in mind that containment fields, another type of force-field, is capable of containing matter and anti-matter which would be pretty tiny in size I would expect. Now, if we follow this theory through to its logical conclusion, that means that it would be possible to erect a lower-level force-field which would allow a transporter beam to penetrate it, but still strong enough to prevent a fully-formed being to escape from it. Be mindful that a transporter beam for personnel works on the quantum level (Source). That would explain how a transporter beam could still penetrate a force-field strong enough that a being could not escape from it.
Now, as to the reason why it is impossible to beam an individual straight out of the brig, it is probably due to brigs using high level force-fields. Considering that brigs are designed to prevent prisoners from escaping, the force-field would probably be set at quite a high level, because prisoners can be crafty and may find a vulnerability!