In 'Beer Bad' Buffy turns into a caveperson by drinking magical beer, but in the next episode she is normal again.
How did this happen?
Also, they didn't seem to stop the bar owners from making more of this magical beer. Why not?
Essentially, the beer only turned a person temporarily into a caveman. The effects were stated to wear off pretty quickly. It seemed to me that the beer was destroyed in the fire near the end, but it is never said exactly what was done about it.
As for proof of the timespan, the transcript states:
Xander: No. No neat. I served them that beer. I served Buffy that beer! Uh, how much beer would you say a person would need to consume before they start seriously questing for fire?
Jack: Relax. It'll wear off in a day or so.
In answer to your question(s)
This is explicitly covered in the original script. Jack, the Grotto's owner indicates that the effects were only temporary.
Jack: ...Relax. It'll wear off in a day or so.
Buffy and her crew don't have the capability of keeping dangerous humans under wraps, relying on the police and local authorities. That said, we do have a few possibilities;
A few episodes later we learn the the US Military have a special holding facility for precisely these sorts of people. Given The Initiative's resources and integration with local law enforcement, it's plausible that the bar's owner would have been taken to that same facility for 'assessment and rehabilitation'.
ETHAN : Well, maybe 'cause you've got no choice. I'm human, you can't kill me. What's a Slayer gonna do to me?
Riley ushers in two large, uniformed, US ARMY MPs. One of the MPs handcuffs Ethan.
RILEY : By the authority of the U.S. Military, you are being taken into custody pending a determination of your status.
The MPs haul Ethan away.
RILEY (to Buffy) : They'll take Mr. Rayne to a secret detention facility in the Nevada desert. I'm sure he'll be rehabilitated in no time.
We learn in Out of Mind, Out of Sight that the FBI keep a reasonable watch on the events in Sunnydale, kidnapping and indoctrinating a girl who's become invisible. It's certainly possible that they "disappeared" him the same way.
There's also the merely mundane. There's a police presence in the final scene and we see several policemen questioning potential witnesses. Given that he knowingly poisoned his patrons, Jack may simply have been arrested and imprisoned for attempted murder or "maliciously administering a poisonous agent", both of which carry hefty prison terms.