How far away from the Gamma Quadrant entrance to the wormhole is the edge of Dominion-controlled territory?
Or is the wormhole inside Dominion territory?
Based on this image from the excellent Star Trek: Star Charts, the Idran system (the closest system to the Gamma Quadrant end of the wormhole) was on the soft edge of Dominion territory, about 250 light years from an area clearly identified as theirs. The species that are first encountered (Tosk/Hunters, Dosi, etc.) are better classed as vassal species rather than members of the Dominion proper.
Ronald D. Moore described the wormhole as "not being in their living room". It certainly looks like their space doesn't extend as far as the wormhole itself, but they have significant interests in the area.
The Dominion does not own the entire Gamma Quadrant. We had explored the GQ for two years before encountering the Dominion, so it's not as though the wormhole opens up in their living room. There are other races in the GQ that are not part of the Dominion and the Ferengi at least have established trade with some of them. When the Dominion told us to stay out of GQ, it was if China told the US to stay out of the Yellow Sea. China is the big boy in this neck of the woods, and you better take their warning seriously, but at the same time we have trading partners and allies there and hey, freedom of the seas and all that.
Here I use two separate lines of reasoning to show that the Dominion border is hundreds or thousands of light years from the wormhole.
According to DS9 "Hippcratic Oath":
Medical Officer's log, stardate 49066.5 Chief O'Brien and I have concluded our bio-survey of Merik Three in the Gamma Quadrant. We're on course back to the wormhole and should arrive at the station two days ahead of schedule.
O'BRIEN: We're picking up a subspace magneton pulse, bearing zero nine zero mark one one five. The Bopak system.
They go to investigate and when they arrive at the planet:
BASHIR: According to our records, Bopak Three is uninhabited. The nearest Dominion outpost is six weeks away and this system is well off the established trade routes in the Gamma Quadrant.
So apparently Merik Three was not a Dominion planet, or one expected to be added to the Dominion soon, or close to the edge of Dominion space. And most people would expect that they were heading in a straight line back to the Gamma Quadrant mouth of the wormhole near Idran.
Most people would deduce that at the time of "Hippocratic Oath" the area around the Gamma Quadrant mouth of the wormhole was not yet ruled by the Dominion. The Dominion had forbidden Starfleet ships from using the wormhole, but didn't yet have fleets stationed there to destroy any Federation ships that came out of the wormhole. So obviously the dominion did not yet rule the area around the Gamma Quadrant mouth of the wormhole.
Then they turned and traveled a few hours or days out of their way to the Bopak system. The Bopak system is said to be "six weeks away" from the nearest Dominion, which should be near the Dominion border, or roughly 35 to 49 days travel away.
O'Brien and Bashir probably traveled less than seven days on their detour to the Bopak system, considering they were running about in a runabout. Thus they should have changed their course toward the Bopak system at a point somewhere between 28 and 56 days travel from the nearest Dominion outpost.
I would also estimate that the travel time between the Gamma Quadrant mouth of the wormhole and Merik Three was less than seven days. Thus both Merik Three and the Gamma Quadrant mouth of the wormhole should be between 21 and 63 days travel from the nearest Dominion outpost to Bopak, and thus to the approximate Dominion border.
Since everyone who watches knows that DS9 should be between about 200 and 400 light years from Earth, and since in "The search, Part 1":
SISKO: Phew. I wonder when that happened? JAKE: What? SISKO: When did I start thinking of this Cardassian monstrosity as home? JAKE: I think it happened last Thursday, around seventeen hundred hours. When you took all this stuff out of storage back on Earth.
So apparently it takes less than seven days to travel from Earth to Bajor. Thus the speed of interstellar travel should be at least 28.57 to 57.14 light years per day.
Since the Gamma Quadrant mouth of the wormhole should be between 21 and 63 days travel from the nearest Dominion outpost to Bopak, and thus to the approximate Dominion border, the Dominion border should be at least 599.97 to 3,599.82 light years from the Gamma Quadrant mouth of the wormhole.
In DS9 Alpha Quadrant ships travel through the wormhole regularly, and the first contact with people coming through the wormhole from the Gamma quadrant comes in the early episode "Captive Pursuit". And in "Move Along Home":
JAKE: Fine. What's going on anyway? SISKO: A Vulcan ship has made contact with a new species called Wadi in the Gamma Quadrant three weeks ago. We're expecting a delegation from their planet. JAKE: Rolling out the red carpet, huh? SISKO: First contact is our most important mission, Jake. It has to be handled delicately. You might say it's sort of like a first date with a girl. You want to show a certain respect for KIRA [OC]: Ops to Sisko
In "Battles Lines" a prison planet is discovered in the Gamma quadrant. The prisoners come from an unnamed other planet somewhere in the Gamma Quadrant.
SISKO: What was your crime? SHEL-LA: The Ennis and the Nol-Ennis are ancient enemies. For generation after generation we fought the same war. The leaders of our planet were unable to mediate a peace. Finally, they sent us all here and told us that we would serve as an example for the rest of civilisation.
In "Vortex" a Klingon ship brings Crodan, a fugitive from the Gamma quadrant, to dS9.
In "Q-less" Vash, a human from the Alpha quadrant, returns after 2 years with Q in the Gamma Quadrant.
KOLOS: (the well-dressed alien) Quark, you obsequious toad, your so-called Gamma Quadrant merchandise had better be legitimate or I'll toss you out the nearest airlock. QUARK: No cause for alarm. Each piece comes complete with a statement of authenticity from Vash, the Federation's foremost expert on the Gamma Quadrant.
So during the first season the Federation encounters several sources of information about the Gamma Quadrant. If you search the transcripts for "Dominion" it is not until the second season that the Dominion is even mentioned. "rules of Acquisition":
ZYREE: The Karemma. PEL: Who's the Karemma? ZYREE: An important power in the Dominion. QUARK: The Dominion? What's that? ZYREE: Let's just say if you want to do business in the Gamma Quadrant, you have to do business with the Dominion.
HANEEK: For eight centuries my people have lived under their rule. We've been forced to work as labourers and servants. We were able to escape when the T-Rogorans were invaded and conquered. ODO: Who conquered them? HANEEK: I don't know their name. I only know they were members of something called the Dominion.
RURIGAN: I was happy living on Yadera Prime, that is, until the Dominion took over and changed our whole way of life. I realised one morning that I didn't really belong there anymore. So I came here, and I used a holo-generator to recreate all the things that I'd lost. I've lived here in my village for nearly thirty years. I've watched its people marry, have children, grow old. And sometimes I even forgot that they were holograms. But it's over. It's over. And I would appreciate it if you'd take me back to Yadera Prime.
And in "The Jem'Hadar":
SISKO: That's Quark. Tell me about the Jem'Hadar. ERIS: They're the most feared soldiers in the Dominion. QUARK: The Dominion? The Ferengi have been trying to open up trade negotiations with the Dominion for almost a year now. We've never heard of the Jem'Hadar. ERIS: Then you've been fortunate. Negotiations with the Dominion can be a dangerous thing.
And none of our characters from DS9 recognize the name of the Dominion until "the Jem'Hadar" despite all the information they have gathered from every possible source about the Gamma quadrant.
No travelers from the Gamma quadrant have even mentioned the Dominion, let alone said where it is or how big it is or anything about its policies. Because our characters don't recognize the name when they hear it.
Until "The Jem'Hadar" and then they merely recognize the name and know about Ferenghi efforts to make contact and open negotiations.
Because most of the Gamma quadrant people they have met live far away from the Dominion and don't know anything about it.
So the border of the Dominion should be hundreds or thousands of light years from the Gamma Quadrant mouth of the wormhole.