ANSWER IN SEVERAL STEPS - long but thorough explanation of what happened and why the trench run was planned
Part 1: The Death Star was designed to be defended from assault by warships
General Dodonna describes that the Death Star was designed to fight against a direct assault by large warships, they did not figure that small one-manned fighters would be a threat to it.
Most likely the Imperial designers figured that the Rebel warships would attempt to approach the Death Star's superlaser disk cannons from the far side of the station (to avoid being fired upon by the superlaser cannon) and travel in low orbit towards the superlaser cannon where the Rebel warships would use their turbolaser cannons to destroy the superlaser disk cannons. I believe the Imperial designers of the station screwed up with the small thermal exhaust port because they would have focused most of their attention on protecting the superlaser disk from attack by warships that could fire turbolaser blasts at the superlaser cannons on the disk. Also remember that there were multiple exhaust ports, this was a small thermal exhaust port that was below a main exhaust port that did not have any design flaws.
Remember that the Death Star had tractor beams to draw in small starcraft, like it did with the Millenium Falcon. My one question that I would ask is why didn't the Death Star crews use their tractor beams to draw the Rebel starfighters? The most likely answer is that the Death Star crew was caught off guard by a surprise attack by Rebel fighters when the Death Star crew was planning for a direct assault by Rebel warships. The Imperial fighters did not have their crews ready - as Darth Vader orders an Imperial officer to get the crews to their fighters. In real-life battles tactical planning errors can cause disasters in terms of timing - in the real-life attack on Pearl Harbor, the American forces on Hawaii had prepared for sabotage, but not fighter attack, their flak guns couldn't do it, the American fighters were not ready for take-off. The Pearl Harbor example shows how tactical planning errors by defenders cause timing delays, and that can cause catastrophe.
The Imperial officer who tells Darth Vader that the Rebel fighters are so small that they are evading the turbolasers sounds exasperated, because as Dodonna indicated, the Imperials were expecting a warship assault.
Part 2: Imperial starfighter defense in combination with large number of turbolasers on the Death Star surface pose serious threat to an above-trench attack
As mentioned above, the exasperated Imperial officer who informed Vader of the Rebel fighters evading the turbolasers, appears to have spoken with the turbolaser crews who told him they can't destroy them on their own. Darth Vader however appears calm because he knows that the Death Star has squadrons of fighters to fight the Rebel fighters.
The turbolaser turret fire in combination with Imperial fighter attacks were presumed to be able to destroy the Rebel fighters. But the Rebels prepared for this in two ways:
1) Diversion: The X-Wings and probably a significant number of the Y-Wings created a diversion by drawing in the TIE Fighters to attack them. Meanwhile a group of Y-Wings broke away from the main group and headed for the Exhaust Port. They approached from a distance away, so as not to make their target seem obvious so as to avoid drawing the TIE Fighters after them.
2) Trench: minimal firepower ability by turbolasers and chasing TIE Fighters: By going into the trench, only a few turbolasers could fire at the fighters in the trench, and only a few Imperial fighters could chase directly behind them. Plus the Imperial fighters would be unable to easily chase them from directly behind would need coordination with the turbolaser gunners in the trench to avoid friendly fire as they chased the Rebel fighters. Plus even if the Imperial fighters entered into the trench to chase the Rebel fighters in front of them they would be easy prey for a group of Rebel fighters following behind them.
Part 3: What happened that made the trench run go as it did
Again returning to the conversation between the Imperial officer and Vader. The Imperial officer tells Vader "We count 30 Rebel ships". During the TIE Fighter attack we see that Rebel fighters are being destroyed by the TIE Fighters. By the time of the trench runs by both the Y-Wings and the X-Wings, there appear to be very few Rebel fighters left even though the main TIE Fighter squadron appears to have been destroyed. Also the Rebel fighters do not know about incoming TIE Fighters because the Death Star appears to be jamming signals being emitted from the TIE Fighters.
The Rebel base was able to detect signals of the main squadron of TIE Fighter's but for some reason does not detect Vader's squadron of his TIE Advanced x1 and two TIE Fighters. Perhaps Vader's squadron signals got mixed up with signals being emitted from the Rebel fighters, as Vader's squadron perhaps stalked their trail. Or the Death Star crew found a way to jam the signals from Vader's squadron from being emitted to the Rebel base. This is feasible because Vader was obviously coordinating with the Death Star's turbolaser crew when the trench's turbolasers would cease firing on the Rebel fighters in the trench when Vader's squadron approached the trench.
That coordination that occurred between Vader's squadron and the Death Star's turbolaser crew was something that Rebels did not plan for. That is why they switched all power to their front deflector screens. One of the younger Y-Wing pilots is completely confused by the sudden stop of the turbolaser fire, but the veteran Y-Wing pilot Gold Five quickly realizes that the most likely reason for this is because the turbolaser crews want to avoid friendly fire on Imperial fighter, and tells the other pilots to stabilize their rear deflector screens. That is when everything goes wrong for the Rebels.
The remaining X-Wings are likely too far away to reach the Y-Wings in the trench in time to save them, as they had been fighting the larger TIE Fighter squadron away from the trench as a diversion. Thus due to the X-Wings' diversion tactic with the main Imperial fighter squadron in combination with Vader's swift recognition of the several Y-Wings breaking off from the main group, the Y-Wings did not gain the defensive fighter cover by the X-Wings that could have easily followed behind any Imperial fighters chasing the Y-Wings in the trench. But the Y-Wings went ahead with the trench run anyway, there was no time for the Y-Wing pilots to spare reorganizing their attack run as the Death Star was nearing its orbital position to fire on Yavin IV.
With no fighter cover by the remaining X-Wings and no time to spare, the Y-Wings went near-suicidal straight ahead. Gold Leader wants room to maneuver from incoming fire from the TIE Fighters by breaking out of formation and gets nervous - Gold Five is focused on reaching the Exhaust Port in time, and tells him to focus and "stay on target". This is probably because Gold Leader's targetting computer may be unable to calculate the direction the torpedo must follow to go into the Exhaust Port if he he continually attempts significant maneuvers in the narrow trench, which would also be dangerous with other fighters near them in the trench (as Vader's wingman found out the hard way).
The X-Wings' trench runs followed much the same. They were all but out of time, there was no time to substantially re-plan the attack. General Dodonna told the Red leader to take ""keep your group out of range for the next run" - i.e. don't head to where the Y-Wings where and where the Imperial fighters are, to keep out of range of the fighters' fire and keep out of range of turbolaser fire so that the fighters would have physical space between them and the three Imperial fighters. Red Leader knew the three Imperial fighters would catch up with them, and told Wedge, Biggs, and Luke to be on the lookout for them.
The big question and almost unjustifiable plot error is that I don't know why Biggs, Wedge, and Luke were unable to go into the trench and draw out Vader's squadron or at least his wingmen - there was no reason for them to make the same mistake that happened with the Y-Wings of the X-Wings being in the wrong place after engaging another group of fighters. The only possible explanation is that Wedge or Biggs thought they saw the fighters in a different location from where they were, and were again a bit too far away. But they were not too far away from Red Leader, as it can be seen that Vader breaks up his squadron's formation as he chases Red Leader's X-Wing, while one of the wingmen TIE Fighters appear to be chasing the other X-Wings as, a TIE Fighter is seen flying below Red Leader's X-Wing after Vader's shoots the fatal blow to Red Leader's starboard engine.
Han Solo's Millenium Falcon struck a lucky blow to one of the TIE Fighters, one of them panicked, the rest is history.
Whew! That was long, but that is what I see as happening and why the trench run occurred and why it occurred the way it did. *Summary:* tactical screw ups by both the Imperials and the Rebels, issues of physical space and location of weaponry on the Death Star (more turbolaser turrets on the surface less in the trench, few Imperial fighters able to fit in the trench to pursue from directly behind), roles of physical distance and timing.
We also didn't see any gun emplacements firing at the fighters from near the exhaust port
They was a very good reason for the guns to not be firing, don't forget who else was flying around in that trench. Do you want to be the guy who accidentally shot Darth Vader's ship ?