The short answer: He found himself in a stressful situation and freaked. Total malfunction.
The long answer: Ash was one of the early prototype synthetics, which were designed to pass as human. These models were covertly placed on long-range vessels with a hidden imperative - acquire any technology or lifeforms with potential profitability to the corporation.
From the moment he saw the alien stuck on Kane's face, Ash had a single objective - secure the creature for study. Unfortunately, this task proved difficult when the thing grew to 7 feet tall and started killing off the crew. Then on top of that, the human crew grew determined to kill off his specimen. Had he been human, Ash could have been described as "extremely frustrated".
At the moments leading up to the scene you describe, Ash has just found that Ripley - already a difficult human to deal with - has just discovered his secret mission and announced her plans to thwart it. At that point, Ash had what (with humans) could best be described as a nervous breakdown. He did the first thing that came to mind - remove the immediate threat. If he could kill Ripley and conceal his involvement, he might be able to salvage the situation. This intermittent inability to deal with extreme stress is what made the A2 model "twitchy" as compared to later models. This was touched on in the second movie:
Bishop: [Puzzled by Ripley's hostile reaction towards him] Is there a problem?
Burke: I'm sorry. I don't know why I didn't even- Ripley's last trip out, the syn-uh- the artificial person malfunctioned.
Ripley: "Malfunctioned"? [Can't believe Burke chose that word]
Burke: [glances at Ripley] There were problems and a-a few deaths were involved.
Bishop: I'm shocked. Was it an older model?
Burke: Yeah, the Hyperdine Systems 120-A2.
Bishop: Well, that explains it then. The A2s always were a bit twitchy. That could never happen now with our behavioral inhibitors. [Looks at Ripley] It is impossible for me to harm or by omission of action, allow to be harmed, a human being.
Unfortunately, Ash had no military programming. He was basically designed for science & subterfuge. Between that and his rapidly deteriorating thought process, he chose a clumsy and inefficient method of killing Ripley - suffocation. He most likely thought that shoving the rolled magazine down her throat and covering the other end would kill her without leaving any obvious signs of foul play. After all, he would be the member of the crew performing an autopsy if it came to that. Her screams brought the other members of the crew, though, and we know how that turned out. As others have mentioned, there was also supposed to be a sexual component to it. As a synthetic, Ash was programmed to understand desire, but could never experience it. They could, however, experience emotion or something very like emotion. It's quite possible that he had unresolved emotions towards Ripley and that influenced events as well.
The white fluid is a liquid substance used to lubricate the insides of synthetics and carry chemical nutrients throughout their body - just like blood in a human. They never explain what exactly it is, but (with a few exceptions) every synthetic in the films and comics had the same characteristic white "blood" to distinguish them from humans.
Both the shaking and the initial seepage from his mouth are the direct results of the malfunction - similar to a human having an epileptic fit or seizure. In Star Trek android lore, it would be similar to a cascade failure. Of course, once the physical assault started, the fluid came from elsewhere as well, with Ripley clawing at him and then seconds later Parker also attacking him.
When I have more time later, I'll see if I can find some quotes or images to support this. This initial answer is based on years of watching the films & reading AvP-related comics series. The notion of the older A2 models going crazy if stressed out was specifically discussed in one of the black-and-white Dark Horse comics, which of course aren't considered canon. A lot of the concepts from them made it into the sequels, however.
EDIT: After searching, I haven't been able to find any additional sources that would be considered official. I added the quote from Aliens that discusses the event, but the actual script for Alien doesn't include much of what made it into the film. In the script, Ash outright attacks Ripley as soon as he finds her in the computer room, and the entire event plays out differently - Ash outright chokes Ripley with his hands and comes across more like a traditional robot without any of the strange behavior.
Wiki sites aren't typically used as answers, but if you'd like to read more about Ash and the synthetics in general, I highly recommend the AvP wiki.