TL;DR: Ultron is to be comparing himself to a god. The available evidences suggests that, if he considers anyone to be a god, it is himself.
Ultron's words:
When Ultron first encounters Vibranium, he says:
Upon this rock, I will build my church.
He's quoting Jesus' words to Peter in The Gospel According to Matthew:
18 And I tell you that you are Peter, and on this rock I will build my church, and the gates of Hades will not overcome it.
- Matthew 16:18; New International Version
His point? As Christians believe Jesus did, Ultron plans to change the world into something more acceptable to himself.
Then he makes reference to the Torah:
Ultron: The human race will have every opportunity to improve.
Pietro: And if they don’t?
Ultron: Ask Noah.
In the Torah, in book of Genesis, YHWH becomes displeased with the human race - except Noah and his family - and wipes out everyone but Noah and his kin.
7 So the Lord said, “I will wipe from the face of the earth the human race I have created—and with them the animals, the birds and the creatures that move along the ground—for I regret that I have made them.” 8 But Noah found favor in the eyes of the Lord.
- Genesis 6:7-8; New International Version
Here, Ultron is again putting himself in the place of god; this time, he is saying that if humanity doesn't change its behavior to endear itself to him, he will destroy it.
And that bring us to the quotes you mentioned in the original edit of this question. First off:
When the earth starts to settle, god throws a stone at it and believe me, he is winding up.
The implication is anything but subtle this time - Ultron is god, and he is literally going to smash the earth with a stone to kill off the humans.
And the other line you mention:
Oh, for god's sake!
I don't see any reason to treat this as anything other than a throwaway line - it's so common today - even among atheists like myself - that it is a meaningless idiom that does nothing more than express one's frustration with an unpleasant surprise.1
But there are other lines in which Ultron paints himself as a god:
At one point, Ultron says he obsesses about meteors as vehicles for extinction:
Ultron: I think a lot about meteors, the purity of them. Boom! The end, start again. The world made clean for the new man to rebuild. I was meant to be new. I was meant to be beautiful. The world would've looked to the sky and seen hope, seen mercy.
Here he all but says that he intended to be worshiped as a god - he thought that humans in his world would see "hope [and] mercy" in him, as believers see hope and mercy in god. Later, he calls back to this line, saying that the Avengers themselves are the meteor he will use to destroy life on earth:
Ultron: Do you see? The beauty of it, the inevitability. You rise, only to fall. You, Avengers, you are my meteor, my swift and terrible sword and the earth will crack with the weight of your failure.
And when Iron Man arrives at Ultron's hideout - which is actually a church - he asks if Tony is going to confess his sins:
FRIDAY: Your man's in the church, boss. I think he's waiting for you. [Stark flies into the church]
Ultron: Come to confess your sins?
I'm certainly not the first person to make this observation; a simple Google search of the terms 'Ultron "god complex' produces over 40,000 results.
Tony Stark's personality echoed in Ultron:
As has been noted in the answers to this question, Tony Stark is an atheist.
As Richard's Valorum's answer says, this has been made clear in the comics:
Cugel's answer is even more relevant, and may demonstrate an intentional hint at what I said in the previous section of this answer.
Tony Stark's bumper sticker:
This is an obvious reference to this popular bumper sticker:
Thus, Tony has replaced god with Jarvis in a tongue-in-cheek manner. Jarvis was Tony's first AI, Ultron is his second. Originally, Tony thought Ultron would be his greatest accomplishment to date - far more impressive than Jarvis. So Tony compared Jarvis to god, then made Ultron, believing that Ultron was better than Jarvis. If Ultron contains vestiges of Tony's own beliefs, it would be natural for Ultron to believe that he himself is more like a god than anything that has ever existed.
An exception to this line of reasoning:
If we're being totally literal here, there is one god that Ultron must believe in: Thor. It would be absurd for Ultron to not believe that Thor exists, because they've actually met:
However, in the MCU, Thor is a "god" in quotation marks, not a God in the sense that YHWH is, so this is basically a semantic argument.
Conclusion:
Ultron's lines about god - except for the one generic idiom - refer to himself. There is no reason to think that he believes in any god(s) besides Thor (who isn't really a god in the MCU). When Ultron references god, he is clearly referring to himself.
Postscript:
I thought it was worth mentioning that Ultron isn't the only one who appropriates Judeo-Christian scripture. When Vision first appears, he says:
I'm not Ultron. I'm not Jarvis. I am... I am.
Which - intentionally or not - echoes Exodus 3:14 of the Torah, where YHWH identifies himself as "I Am":
And God said unto Moses, I AM THAT I AM: and he said, Thus shalt thou say unto the children of Israel, I AM hath sent me unto you.
- Exodus 3:14; King James Version
1 After all, Joss Whedon - the director of Age of Ultron - is an atheist, and he mentioned god in a similarly idiomatic manner in describing the Ultron character:
He's always trying to destroy the Avengers, goddamn it, he's got a bee in his bonnet. He's not a happy guy, which means he's an interesting guy. He's got pain. And the way that manifests is not going to be standard robot stuff.
- Source
And he used the actual phrase "for god's sake" in another interview about Age of Ultron:
What I love is we have the opportunity, because we went in with this mission statement, to shoot the Hulk like a character in a movie, and not like a ‘Look what we’ve got!’ We have ‘overs’ – blurry ‘Hulk’s over there!' bits, very quick shots. Everything isn’t, ‘We built the Hulk so for god’s sake you’re going to watch him in this long take, full-frame, the whole time!’ We really got to make him one of the characters in the movie. And that was a gift.
- Source
