There are (obviously) no direct answers to any of your questions, but here's my take on them, in order:
Why does Penguin consider Gordon a friend?
He doesn't. Penguin treats everyone with the same submissive-yet-condescending attitude in any situation where he has the advantage (His occasional interactions with Fish while under Maroni's protection, for example), as he does in this case. It's important to remember that Penguin never deals with someone who can hurt him unless he wants something, and Penguin wants something from Jim: favours.
What Penguin is doing in this episode is establishing himself as someone Jim can trust. Penguin knows that Jim wants to reform the GCPD, and he knows that Jim will be going up against Falcone in the process. By establishing himself as Jim's go-to guy, he can milk favours out of the rising star.
This is exactly what happens in season 1 episode 18, "Everyone Has a Cobblepot". Jim needs information that only someone in Falcone's network can provide, and wouldn't you know it: Jim knows a guy deep in Falcone's organization who's willing to squeal, in exchange for a favour.
In light of that, Penguin's attitude towards Jim in "Welcome Back, Jim Gordon", specifically when he says:
Penguin: Friends don't owe friends, Jim. They do favours because they want to.
Suggests a sales tactic. The message to Jim is "You can have this one for free, but then you start owing me."
Why does Gordon think Penguin would help? Why not ask Harvey Dent?
I'm lumping these together because the answer is mostly the same: Gordon is desperate, and in a very tough situation. He can't go to Dent for one big reason, even beyond the fact that Dent has proven untrustworthy: there's very little that Dent, a legal official, can do in this situation.
Gordon tried the up-and-up approach earlier, when he raided the stash house, and that failed spectacularly. The "Narcotics investigation" alibi is a good one, and gives Flass and his guys a lot of room to move; literally any drug-related activity can be attributed to this investigation, and the kickbacks going up the chain ensure that nobody hears any different. What's an ADA going to do about any of that?
No, Gordon needs to go Dark Side for this. For that, he needs a criminal he can trust. In other situations he might go to Fish, but obviously that's not going to work for him. He probably feels like he can't go to anyone in the Falcone family, or he's too proud to after his last run-in with them.
That leaves the Maroni family, but he can't just walk into Maroni's restaurant and ask for favours. Penguin is a lowlife in Maroni's gang (as far as Jim knows), with whom Jim has a relationship. He doesn't know if Penguin will help, but he has no other options.
Why was Gordon so surprised at the end of the episode?
My interpretation of that scene was that Gordon had convinced himself that the ends justified the means, but seeing the results of his actions directly,
the formerly-untouchable Delaware quaking in his boots, literally begging Gordon not to hurt his family,
means Gordon can't ignore it any more. I interpreted the scene not as surprise at Penguin's actions, but as the realization that he's become as bad as the people he's trying to take down.