I know what you mean. That part didn't sit quite right with me either. I had to think about it for a while.
Being "not of this world": Bronagh is a fairy, and the fairies/Deenashee are explicitly separate from humans, except for Saoirse, who's half human. The Deenashee are diminished and trapped by a world that no longer believes in them. When Saoirse frees them, they look larger and more alive, and they immediately headed for their true home, another world called Tír na nÓg. For some reason, before this point it was all right for fairies to live in the human world and even intermarry with humans. When Saoirse sings her song, that frees them, but it also means that they must leave our world, and that includes Bronagh. Maybe that's why Bronagh never sang that way. She wanted to stay with her family.
Why she started a family with Conor: Her first child was human, so maybe she thought there was a good chance all of her children would be the same way. But her daughter turned out to be a selkie, and for Saoirse's safety Bronagh apparently had to give birth in the water. For whatever reason, after that, she could not come back to the land ever again. In the various versions of the selkie tale, when the selkie makes her inevitable return to the sea, she never returns. In that way, the movie's following the example of the folktale.
So I don't think she was being cruel at all. The whole story is a metaphor centered around a family with a deceased parent. Fairies are connected with ghosts in a lot of folklore. Now, when Bronagh reappears, her eyes are shut. She's asleep while she's interacting with Saoirse. Sleep is often a euphemism for death. It's only after Ben starts calling her that she wakes up. In short, my personal explanation is that she's dead, at least to the human world. However, because she's a Deenashee, she's still alive in a way, although diminished like all the other fairies. Thus, she must move on to Tir na NOg.