It sounds like you may be crossing two stories.
HP Lovecraft's 'From Beyond' involves a scientist creating a device that stimulate dormant senses in the human mind, allowing them to see thing from beyond this plane of existence. Once able to be seen, they, too, can see those who see them, and things go downhill from there. The creatures are very amorphous, and creep out the protagonist as he notices them overlapping and combining at times. You can read it on Wikisource, for free. (It's long past copyright.)
I'm guessing the geometric shape aspect comes from The Hounds of Tindalos by Frank Belknap Long, which involves traveling back in time (mentally), and creatures that chase the protagonist back through 'strange geometries', and, specifically, can enter the world thru angles. (The eventual victim tries to protect himself by eliminating all the angles in the room he stays in, until they get tired of the pursuit.) It, too, is on Wikisource, but I'm not sure of it's copyright status, so I can't guarantee it will last.
Both are part of the Cthulhu mythos, although the first was written by Lovecraft, and the second by Long; you may well have encountered them in the same collection, tho.