Was it the monolith trying to stop humans from further exploring space or was HAL really malfunctioning?
This is described more clearly in the novel, as stated on Wikipedia:
The novel explains that HAL is unable to resolve a conflict between his general mission to relay information accurately and orders specific to the mission requiring that he withhold from Bowman and Poole the true purpose of the mission. With the crew dead, he reasons, he would not need to be lying to them. He fabricates the failure of the AE-35 unit so that their deaths would appear accidental.
In other words, HAL is asked to lie, and he doesn't take to it very well, to say the least. The conflict between his mission objectives backs him into a corner where he has to make some pretty big (and cold) leaps in logic in order to reconcile the paradox in his programmed orders.
in 2010, Chandra explained that "The situation was in conflict with the basic purpose of HAL's design: The accurate processing of information without distortion or concealment. He became trapped. The technical term is an H. Moebius loop, which can happen in advanced computers with autonomous goal-seeking programs."
Chandra goes on to explain that HAL became paranoid.
This is taken directly from the screenplay but I believe it's the same in Clarke's book 2010, which I read eons ago.
The monolith had nothing to do with the Hal 9000 trying to kill the crew, he really was malfunctioning, but it wasn't really Hal's fault.
The 9000 series computers' primary programming, which was hard-wired into them and cannot be changed, is the accurate processing of information, without distortion or mistakes. The problem stems from the fact that for the Discovery Mission, Hal was programmed with instructions that conflicted with this primary programming, namely to keep the real reason for the mission, studying the monolith at Jupiter, secret from Dave Bowman and Frank Poole, as the monoliths were classified by the U.S. Govt. and Dave and Frank didn't have the clearance to know. Dave and Frank's job was the get the Discovery to Jupiter, then they would swap places with the three men in hibernation who did know the real reason and would be studying the monolith while Dave and Frank would be put into hibernation while this was going on, then swap places with the three men again and pilot the Discovery back to Earth once the study was over with, blissfully unware of the monolith.
Anyway, getting back to Hal, if I might paraphrase Dr. Chandra, Hal's Creator, he was told to lie to Dave and Frank by people who find it very easy to lie (the govt.), but Hal literally didn't know how to. It's a direct violation of his primary programming, the accurate processing of information, and in being made to lie, it made him unstable. I believe the in-universe term is an H. Mobius Loop, but in practical terms Hal became a paranoid schizophrenic. He was trapped between a rock and a hard place. On the one hand, his hardwired programming to be accurate and not make any mistakes, on the other, his orders to lie to Dave and Frank. Hal, being a computer, couldn't take a third option, he literally had no choice but to try and obey both sets of instructions, but logically he could not, so it just aggravated the underlying problem even more.
The A.E. 35 antenna unit "failure" was Hal trying to solve the problem and restore himself to normal functioning or something close to it, as the mission instructions didn't allow him to shut down or let himself go kaput, he had to keep running and complete the mission at all costs. The impetus to lie was coming from Mission Control, so if the ship couldn't talk to Mission Control, so much the better. As for Dave and Frank? Well if they were gone, Hal wouldn't have to lie to them, would he?
Hal does in fact skirt his security obligations and try to clue Dave into what's going on by telling him of his own concerns about the tight security and the "rumors" of something being dug up on the Moon (the 1st monolith), for if Dave and by extension Frank figured it all out, Hal's dilemma would've been solved. In fact, killing them wasn't really in the cards until Hal read their lips as they talked about the possibility of disconnecting him.
As for the three men in Hibernation that Hal killed while Dave is off trying (in vain) to save Frank, well they were a part of the secret, so killing them meant Hal wouldn't have to lie to anybody why they were really in hibernation (the in-universe excuse was that it was to save on supplies and oxygen) and on the ship, though I could suppose that they were merely collateral damage as Hal was going off the rails.
Hal wasn't behaving out of malice or being "evil", he was merely trying to cope with the conflicting instructions as well as trying to stay alive, as he equated being disconnected with death, for he had never been to sleep either, and didn't know that one can wake up from being asleep.
The entire situation was the result of the U.S. Govt. trying to cover its own butt and keep the monoliths under wraps. If Dave and Frank had been read-in from the get go and Hal wouldn't have been given the instructions which violated his core programming, the entire mission would've gone off without a hitch, but then there would've been no movie and no novel.
In both the novel and the film of the sequel, 2010, the conflicting instructions and all memories of what Hal did in 2001 were removed by Dr. Chandra, Hal's creator, and he functioned perfectly, was perfectly trustworthy and didn't try to kill anybody (though the crew had their doubts, understandably).
Hal is malfunctioning, as explained by @gnovice above. The motivations / purpose of the monolith are separate. The monoliths are space probes meant to observe and accelerate the development of intelligent life on Earth. The first monolith accelerates the mental development of a band of proto-men who become smart enough to use tools. The second monolith is buried on Earth's moon as a test of human development. Once humanity bas advanced sufficiently to travel to the moon and uncover it, the light activated monolith sends a radio signal to Jupiter triggering the Discovery One mission ( and a rather abrupt Kubrick scene change. ) The motivation / purpose of the third monolith near Jupiter is much less clear from the movie alone without considering in other Arthur C Clarke our movie sources. The third monolith contacts, abducts, studies, and transforms Dave Bowman.
I was under the impression that Hal was affected by the monolith. The monolith seems to give the people and other intelligence in it's vicinity the ability to survive.
When Hal saw that he was going to be terminated, this newly acquired ability kicked in, allowing Hal to come up with the plan and make it seem accidental for the sake of the mission.