What happens to the worlds that are within Federation Space that decline to join the UFP? How are they treated?
I think the simple answer is: They got to keep their space.
I have no official source to back this up, but if we look at the Federation, then the most plausible way to deal with worlds which are enclosed by Federation space is to give them their space.
E.g. imagine a world within Federation-Space boundaries, if that world does not want to be part of the Federation, it would most likely get a certain "private" space assigned to it and maybe further privileges (like fly-through Federation space to other stars etc.).
From there the diplomatic relations are pretty straight forward like with every other world outside their borders.
Another example: In the 4th-season Next Generation episode "First Contact", Chancellor Durken, a planetary leader, asks Captain Picard:
And if I should tell you to leave, and never return to my world?
We will leave, and never return.
It's strongly implied (in that episode and elsewhere) that this is Federation / Starfleet policy.
To use a real world analogy, Switzerland is entirely enveloped by the European Union. There is no animosity between the EU and the Swiss. They are regular trade partners and the borders are open. The Swiss have signed a number of treaties which harmonize their laws and regulations with the EU yet they retain their independence, constitution and currency.
This is made easier because of the structure of the European Union. Many things we think are part of the EU (for example, who gets to use the Euro) are actually other treaties which allows non-EU members to participate (and EU members to not participate). It's complicated.
I would expect the Federation to act much the same way. Some members, like the UK, would never consider being military enemies but are always grumbling about the rules and stumping for exceptions because they are a unique, and rich, snowflake (no Euro, no free travel). Some non-members, like the Swiss, would enjoy economically playing nice nice with the Federation but would not want to participate in their military or politics. Some non-members would use Federation credits because their economy is too small to bother with their own currency.
On the flip side, the Federation would politically and economically pressure its enveloped neighbors to stop doing things it doesn't like. Another real world example from the Swiss and EU is world pressure on the Swiss to get rid of secret Swiss bank accounts as places for people to hide their money. Instead of rolling tanks in, the EU and US threatened to move their money out. "Do this, or we won't bank with you anymore and collapse your economy". I would expect the Federation to deal with its enveloped neighbors in much the same way, using trade pressure rather than starships.
This is just an analogy, and I don't have canon reference to back it up, but I feel its a good one. The Federation is cobbled together out of different cultures and species with their own history, traditions and laws just like the EU.
Depending on the outlook, nothing, but a good example of this is the Maquis who were Federation born citizens but rebelled against the Federation and started their own independent state and were almost at the point of formally declaring this to the Federation, until The Dominion wiped them all out.
A quote from the Maquis wiki.
The Maquis considered themselves an independent state and were going to formally declare their separation from the Federation, but these plans never materialized, because they were eventually defeated by one of Cardassia's prime allies, the Dominion and its powerful army of Jem'Hadar
How they were treated was generally bad (albeit they did start deploying guerilla tactics against the Cardassians and Federation), even though technically outside of the jurisdiction from the UFP, the Federation generally treated them as traitors and they were enforced under Federation law. They were free game to every other alien race out there, The Dominion destroyed them for their Cardassian pals because of how the Cardassians viewed them as terrorists.
This is addressed in the Star Trek: Star Charts factbook.
Planets that are deemed unsuitable to join the Federation (or simply choose not to do so) appear to be placed into enclaves, surrounded by Federation space, but not part of the Federation proper.
I think we can be reasonably sure that the Federation will attempt to maintain positive diplomatic relations with these non-aligned planets and continue to evangelise the positive benefits of joining.
On the STTOS episode "Journey to Babel" a secret agent from a "neutral" planet, who was surgically altered to appear Andorian, set a trap for the Enterprise as it carried diplomats to the planet Babel where negotiations between Federation planets were to take place.
Also note frequent episodes in all of the series in which the Federation makes first contact with alien civilizations. In these episodes, the Federation treats the people on these planets as humanely and respectfully as possible.
It appears that the Federation marks out and lays claim to all space it travels in until it reaches another alien species that also lays claim to an exclusive area of space such as the Klingon Empire or Romulan Star Empire in which case a treaty is negotiated designating an agreed to amount of space as belonging to the alien government which is surrounded by a buffer zone claimed by neither party (designated a Neutral Zone) serving as a barrier between the two regions that neither party are permitted to enter (the distance across the neutral zone presumably being negotiated and agreed to by both parties in the treaty).
Certain worlds are not permitted to join the Federation due to lacking in technological development (pre-warp) or on account of certain conditions existing on the world that is incompatible with the Federation government such as the Orion Syndicate’s participation in the slave trade or the Ferengi’s obsession with profit and acquisition of wealth, who are permitted relative freedom of movement within Federation Space though they are not actually members of the Federation. Another condition required by the Federation is that the planet be unified (or at least) have a universally recognized governmental organization who can speak and act on behalf of the planet as a whole with the Federation during negotiation prior to membership and through such governmental organization an Ambassador to the Federation Council could later be chosen to represent the whole planet in the Federation Legislature in San Francisco. Such worlds are considered neutral worlds. Other worlds that for various reasons have been rejected admission to the Federation would share equivalent status as a neutral world.
Depending on the circumstances, a neutral world may be completely avoided by the Federation as “off limits”; may participate in trade and act in a manner similar to a member world in varying degrees of contact but without the protection of Starfleet against any alien incursion or assistance with any domestic conflict (the Orion Syndicate, Ferengi); or may be studied in secret such as the Vulcan relatives discovered on Mintaka III (ST:TNG “Who Watches the Watchers”).
As yet, there have been no instances of a member world of the Federation attempting to leave the Federation after having joined that I am aware of, and so whether the Federation is a union by continued voluntary consent of its members (in practice) or would be held together by force of arms enforced by the phasers and photon torpedoes of Starfleet, has as of yet to be tested on any official Star Trek series.