First of all, it's important to note that, despite the similarities in name, angreal and sa'angreal are very different from ter'angreal.
The first two are only used to enhance the amount of the One Power someone can channel; the distinction between the two is pretty blurry, but essentially boils down to "sa'angreal are just souped-up angreal". And yes, as you surmise, every one we ever see, or even hear of, in the story is keyed to either saidin or saidar. It's possible but highly unlikely that any of them can be used by both.
One example of why we believe this is true, which we learn a bit about in The Shadow Rising, are the statues Rand find in Rhuidean. These statues are actually ter'angreal, but they are tied to a pair of sa'angreal - one for men and one for woman. Specifically, they're access keys to gigantic statues, called the Choedan Kal. (This female one is the statue that starts "glowing" on the Sea Folk's island in The Great Hunt.)
Note that during his fight with Asmodean, both statues are present in Rhuidean, but Rand and Asmo fight over possession of the male. The fact that that two statues were built, one specifically male and one specifically female, combined with the fact that neither Rand nor Asmodean attempted to use the second, seems to imply that this male/female distinction is hard-coded into the process of making an angreal.
ter'angreal, on the other hand, are a totally different question. These are basically "magic items", imbued with some built-in portion of the One Power at creation. Who can use them is entirely up to the person who created it -- in some cases, they are tied to one gender or the other, in some cases either gender can use them. In some cases, you need to channel to use a ter'angreal while others can be used by anyone who knows the process.
As an example, one ter'angreal we see and hear a lot about from very early on is the Aes Sedai Oath Rod. It's probably obvious from the beginning that this is a ter'angreal and that you need to channel to activate it. We eventually learn that this ter'angreal is not gender-specific, the Aes Sedai just have no idea what its original purpose is.
It's actually a "binder", and was used to punish Age of Legends channelers who broke the law.