In Infinity Gauntlet (1991) comics, it was hinted in this page:
Who are they?
The Elders of the Universe are intergalactic senior citizens, some billions of years old, of now defunct civilizations who pass their time with an obsession based on their previous cultural predispositions. In most cases, they are the last survivors of their civilizations.
All of them are wielders of the Power Cosmic (or Power Primordial as a few of them call it) and utilize it in various forms, enhancing their powers and abilities. Some are masters of technology, other utilize personal abilities in their quest for personal fulfillment.
For a time, they were even banished from the realm of Death and were not just functionally immortal (a side effect of their interaction with the Power Cosmic) but were literally unable to enter the realm of Death.
Using your graphic:
The Soul Gem was in the possession of the In-Betweener, an entity who worked for Master Order and Lord Chaos, abstract entities and servants of the Living Tribunal. He is the only entity on this list who is not an Elder of the Universe.
The Mind Gem was in the possession of the Grandmaster which he used to enhances his already prodigious intellect further unbalancing games of chance because he had even greater mental capacity for predicting the outcome. Thanos left him in a catatonic state.
The Reality Gem was in the possession of the Collector, who let it sit in his collection doing nothing other than to say he possessed it. Thanos traded it for the Runner who had been de-aged and the Collector threatened to place him in his collection...
The Space Gem was in the possession of the Runner, an Elder obsessed with being the fastest entity in the Universe. In his hands he used it to run/teleport, furthering his ambitions as the definitive speedster in the Marvel Universe.
The Power Gem was in the possession of the Champion of the Universe, whose avocation is being the most powerful fighter in the known Universe used its limitless energy stores to give him unending stamina and hitting power in battle.
The story Thanos Quest documents Thanos' acquisition of all the gems, and who he got them from.
Elders of the Universe are each the last surviving members of their species, each generally obsessed with a single hobby which they get their name from. The Collector has been portrayed in Guardians of the Galaxy and got a cameo in Thor 2, and The Grandmaster will be in Thor 3, but the other Elders are relatively minor characters, appearing only in a few obscure stories.
The In-Betweener is the only one of the gem-bearers(?) who is not an Elder. He is a cosmic entity that represents duality, servant of other cosmic entities, Lord Chaos and Master Order, and the physical embodiment of the balance between them.