UPDATE: Since you've rewritten your question, I'll try to address it a bit better. I'll leave my original answer below.
To the best of my knowledge, no, vampire teeth are never noted as having any magical properties (piercing or otherwise).
Vampires don't seem to use their teeth as weapons (except to drain their prey), nor as tools. I can't recall any case off-hand of a vampire biting something other than human(-ish) flesh. I can't recall a case of a vampire trying to bit a demon either, let alone one with impenetrable flesh - is there even such a thing in the Buffyverse?
I may be forgetting something, but given how strong and tough Hamilton is, I doubt a vampire on the show has bitten into anything tougher. How many things on the show were tougher? Adam? Glory? The demon the mayor becomes? Illyria pre-depowering? Ubervamps? Caleb? None of those is bitten.
Even if you could find some solid evidence, I doubt the show has enough consistency to draw any real conclusions. It seems unlikely to me the writers sat down to calculate vampire jaw strength, and instead gave them the strength required by the plot.
Marcus Hamilton is clearly not invincible, since Angel snaps his neck after being powered up with his blood.
Yes, Marcus Hamilton takes punches from Angel and (de-powered) Illyria without flinching. (Actually Illyria only lands one punch, Hamilton dodges another and blocks a kick.) He's very strong and very tough. But lets put this into perspective - Angel is thrown by Hamilton through a pillar without suffering serious injury.
Furthermore, at no point in the fight does Angel use any sort of weapon - it's a brawl. Likewise with Illyria (who only lands a single punch). Marcus Hamilton isn't hit with anything sharp or pointy besides Angel's teeth. So we have no frame of reference to judge how hard it would be to penetrate his skin.
Given this, I see no reason to think Angel's bite would need to be magical to penetrate Hamilton's skin. Keep in mind that penetrating skin isn't just about the amount of force, but also about the area over which it is spread - pointy teeth focus all of Angel's jaw strength into a small area.
Bolstered by Hamilton's blood, Angel is finally strong enough to hurt Hamilton (and possibly Hamilton is weakened), and eventually lands a solid blow to his head that snaps neck.
You might also compare Angel's fight with Hamilton to Buffy's fight with the ubervamp in Buffy S7E11 "Showtime". The ubervamp is tough enough that Buffy fails to stake it - she doesn't strike hard enough to penetrate the ubervamp's rib cage (though she does penetrate its skin). However, she is later able to wrap some (entirely mundane) wire around it's throat and apply enough force to behead it. It isn't just about relative strength, it's also about how that strength is applied.
