Subspace is bizarre phenomenon used for many things in the Star trek universe, most notably communication. Subspace transmissions travel faster than light (but still at a finite speed), allowing almost instant communication between ships that are "nearby" in interstellar terms. While the physics of subspace technology is left deliberately vague, it is established that subspace communications can operate on different "frequencies", presumably to parallel how radio communications work in reality. The famous line "Hailing frequencies open, sir" is a good example. Therefore, the subspace frequencies Nero speaks of are probably the frequency that Earth's defense systems use to coordinate and communicate.
This is confirmed by the Memory Alpha article:
Border protection grids were security systems utilized by the United
Federation of Planets to protect individual member worlds. Each grid
utilized specific subspace frequencies which, if known, could be used
to bypass the grid.
While the details of exactly what a "Security System" is in this case, there certainly are a lot of possibilities. Obvious it refers to some type of technology designed to protect the earth from invasion, whether it be an armada of ships, automated weapons platforms, an enormous deflector shield, or any of the various other planetary defense systems utilized in the Star Trek universe.
Whatever these defenses were, it seems very hard to believe that they would have honestly posed a challenge to Nero's seemingly invincible technology. It was pretty much an excuse for Nero to torture Pike for dramatic purposes.