The Restricted Section of the library of Hogwarts contained dangerous books. Why wasn't it locked or protected by some kind of spell? Certainly, Filtch (or Madam Pince,) wasn't a good protection against a rogue wizard who could easily stun or kill him before he could call skilled wizards.
If you mean protection against a kid (like Harry) picking up a dangerous book - and doing some damage to themselves, like Harry did in the Christmas of First year - then it's the same reason there's no electric fence around Forbidden Forest, despite dangerous animals living there. All you got was Dumbledore's warning at the starting feast.
If someone is bound and determined to do damage to themselves in a Wizarding world, they easily can. This means that mollycoddling children and nannying them is beyond useless - you need to teach them to follow the rules if those rules are clear about the danger.
Also, remember that Wizarding society is modeled after late medieval/early modern Muggle society. Back then, kids were treated more like small adults than the way they are in modern societies.
If you mean dangerous wizards outside school, the school has multiple wards. No stranger danger there. And if a dangerous skilled wizard enters the school, they probably are already at a level where needing some book from a school library is well beneath their knowledge.
This is a school library we're talking about, not the dungeons of the Ministry or Voldemort's private study. Although some of the books contain advanced spells (and in some cases dark magic spells) the books weren't kept under lock and key, they were simply restricted from junior wizards. The main reason seems to have been to prevent them from harming themselves or others through sheer youthful incompetence:
Harry wandered over to the Restricted Section. He had been wondering for a while if Flamel wasn’t somewhere in there. Unfortunately, you needed a specially signed note from one of the teachers to look in any of the restricted books and he knew he’d never get one. These were the books containing powerful Dark Magic never taught at Hogwarts and only read by older students studying advanced Defence Against the Dark Arts. - Harry Potter and the Philosopher's Stone
Madam Pince seems happy enough to provide Harry with full access on the signature of Professor Minerva McGonagall (in Goblet of Fire) and both Tom Riddle and Hermione Granger spend a considerable amount of time in the restricted section researching dark magic, specifically Horcruxes.
Note that in the books, the sole protection is a rope. It's clear that students aren't intended to be permanently removed from them, the barrier is largely symbolic:
The Restricted Section was right at the back of the library. Stepping carefully over the rope which separated these books from the rest of the library, he held up his lamp to read the titles.- Harry Potter and the Philosopher's Stone
In my opinion the restricted section of the library was already as protected as it should be
It was inside Hogwarts, which already makes it protected enough from outside wizards
Moreover, it is not the Department of Mysteries. Although they contained dangerous information, they were still books. To be actually able to cause any damage using their information, a student would have to be determined enough to break into the section, determined enough to learn the spells and powerful enough to actually use them to good effect
The library would probably come under the jurisdiction of Madam Pince the librarian, and although we do not have much information about her abilities, she was an actual witch with magical powers, and probably more qualified than Filch, which makes it more probable that she had put as much protections over the restricted section as practically feasible (from whatever mention we have seen of her in the books, we can get the feeling that she takes her job seriously)
To be fair, we do see the restricted section in the library being broken into, which itself shows that the protections were not enough, which does justify this question. However, we should also keep in mind that the break in (I am referring to Harry in book 1, their getting hands on Most Potente Potions in Book 2 was through the proper channels and has nothing to do with the security surrounding the Restricted Section) was a special case. Students were not allowed to be outside their dormitories after dark, and Harry was able to snoop around only because he had an invisibility cloak. From what we know of invisibility cloaks, a student in Hogwarts having one is a rare case in itself. Also, harry tends to do things that he thinks are right, regardless of whether there is any concrete proof of if they are or not. The only reason he broke into the restricted section was because he got a feeling he would find Nicolas Flammel in one of the books, which may or may not have been true.
Thus all these special events came together to make the break-in possible. This was probably a sort of one-off incident, and the restricted section is protected enough for general cases.