Some of it is left a little ambiguous, but here is the rough breakdown.
After The Undertaking at the end of Season 1, Queen Consolidated loses much of its value as Starling City (and presumably much of the world) has little desire to deal with a company owned by one of the people directly affiliated with The Undertaking. As such, the value of the company generally plummets.
As is frequent for a public company, Oliver and the Queen family does not actually own more than half of the shares of the company - they own majority share, but not enough to necessarily maintain absolute majority. (Note, it's possible that they did, but that more shares were released in order to finance the company. It's never stated either way, that I can recall.)
Enter Isabel Rochev, who works with Stellmoor International. They have enough capital to purchase enough shares to become Oliver's direct equal in the company.
Unfortunately, in a moment of panic and seeming need to delegate authority, Oliver signs what he intends to be temporary authority over to Isabel, who uses the power to get the board to give her full permanent control over the company.
At this point, things get very hand-wavey. Isabel begins somehow decreasing the Queen's value, presumably by working to lower the value of the company's stocks, and since most everything associated with the Queen family is apparently actually owned by Queen Consolidated, they lose much of everything they have - including the building used by Verdant. There is an attempt by Oliver to get Thea to sign some papers to help prevent it, but Thea refuses and Oliver is unable to stop the fortune from effectively drying up.
This brings us to the end of Season 2, which sees another disaster in Starling City, as Slade's men attack the city. While the attacks cannot be linked to the Queen family by the public, the attacks would have an effect on lowering the desire of anyone wanting to live in, or work with a company based in Starling City (this is all reflected in Ray's speech to the board members).
With the support from the board members, and more money behind him, Ray Palmer is able to purchase control of Queen Consolidated.