Some do, and some... might not.
In alternate universes, such as:
- Earth-838 from Doctor Strange in the Multiverse of Madness
- The universe showed in What If...? Episode 2 "What If... T'Challa became Star-Lord?"
- The universe showed in What If...? Episode 8 "What If... Ultron Won?"
In addition, in What If...? Episode 9 "What If...? the Watcher Broke his Oath?", a multiversal team of heroes is tasked with using the Infinity Crusher to destroy the Infinity Stones held by Ultron. However, it doesn't work, and Ultron says:
ULTRON: I’m honestly surprised. Didn’t the Watcher warn you? Every universe is different, each one just a fraction unique. Thus the Infinity Stones are unique.
This implies that each universe very well could have its own version of Infinity Stones, but they could be ever-so-slightly different on a universe-to-universe basis. However, it doesn't necessarily confirm that each one has Infinity Stones. It is enough, however, to surmise that a lot of the universes do. However, we don't have explicit confirmation that they ARE in every universe.