Hit 'em where it hurts
As pointed out by Dima, Oberyn pierced Gregor in the gap underneath the arm:
Clegane lifted his own shield against the glare. Prince Oberyn’s spear flashed like lightning and found the gap in the heavy plate, the joint under the arm. The point punched through mail and boiled leather. Gregor gave a choked grunt as the Dornishman twisted his spear and yanked it free.
-A Song of Ice and Fire: A Storm of Swords, Part 2: Blood and Gold - Chapter 71 (Tyrion X).
Pop, pop, bang
Oberyn does later (at the end of the fight) pierce through Gregor's chest-plate armour, but he puts an incredible amount of force behind it:
Slowly, ponderously, he [Gregor] rolled onto his back.
The Dornishman flung away his ruined shield, grasped the spear in both hands, and sauntered away. Behind him the Mountain let out a groan, and pushed himself onto an elbow. Oberyn whirled cat-quick, and ran at his fallen foe. “EEEEELLLLLLIIIIIAAAAA!” he screamed, as he drove the spear down with the whole weight of his body behind it. The crack of the ashwood shaft snapping was almost as sweet a sound as Cersei’s wail of fury, ...
-A Song of Ice and Fire: A Storm of Swords, Part 2: Blood and Gold - Chapter 71 (Tyrion X).
I wouldn't touch him with an eight foot pole
Remember, this is not any ordinary old spear:
The spear was turned ash eight feet long, the shaft smooth, thick, and heavy. The last two feet of that was steel: a slender leaf-shaped spearhead narrowing to a wicked spike. The edges looked sharp enough to shave with.
...
“You will have no cause for complaint. Though Ser Gregor may. However thick his plate, there will be gaps at the joints. Inside the elbow and knee, beneath the arms... I will find a place to tickle him, I promise you.”
-A Song of Ice and Fire: A Storm of Swords, Part 2: Blood and Gold - Chapter 71 (Tyrion X).
Oberyn is now a Faceless Man
As to the smashing of the face, Ser Gregor "The Mountain" Clegane was a humongous man, he has a tremendous body mass as well as strength:
Gregor Clegane is well known for his size, cruelty, and prowess in battle. Gregor is extremely tall (he is close to eight feet in height). He has "massive shoulders and arms thick as the trunk of small trees." Gregor weighs over thirty stone (420 lbs), nearly all of it muscle, making him near inhumanly strong. Gregor's strength allows him to wield a six-foot, two-handed greatsword with just one hand, giving him enormous reach while still wielding a shield. Such is the power of Gregor's strength that he has been known to hack men in half with just a single blow.1
Jaime says of Cregor's strength:
Robert had been stronger than him [Jaime], to be sure. The White Bull Gerold Hightower as well, in his heyday, and Ser Arthur Dayne. Amongst the living, Greatjon Umber was stronger, Strongboar of Crakehall most likely, both Cleganes for a certainty. The Mountain's strength was like nothing human.
-A Song of Ice and Fire: A Storm of Swords, Part 1 - Steel and Snow, Chapter 21 ([Jaime III][3]).
So when Gregor shoves his hands and the weight of his whole body onto a man's face, it is not implausible for him to do the damage that he did. Albeit there is slightly less of a demolition of Oberyn's entire head in the books, nevertheless the face was crushed:
He thrust his free hand into Oberyn’s unprotected face, pushing steel fingers into his eyes. “Then I raped her.” Clegane slammed his fist into the Dornishman’s mouth, making splinters of his teeth. “Then I smashed her fucking head in. Like this.” As he drew back his huge fist, the blood on his gauntlet seemed to smoke in the cold dawn air. There was a sickening crunch.
-A Song of Ice and Fire: A Storm of Swords, Part 2: Blood and Gold - Chapter 71 (Tyrion X).
[All emphases added by me]
1. http://awoiaf.westeros.org/index.php/Gregor_Clegane#Character_and_Appearance