This is the Archives of Anthropos series. The books were written in open imitation of the style of the Chronicles of Narnia, including having plots that serve as Christian allegories. The first two books, The Tower of Geburah (1978) and The Iron Sceptre (1981)--the only ones that existed at the time--were the favorite books of one of my friends in elementary school. (However, the school librarians had never heard of them.)
The book you are remembering specifically (with the fat, unpleasant cousin brought along--an element borrowed from The Voyage of the Dawn Treader) is the second one written, The Iron Sceptre.
Goodreads on The Iron Sceptre:
Anthropos is in danger. So Wesley, Kurt and Lisa are once again magically transported from their home in Winnipeg to help save this threatened land. The Iron Sceptre, the source of King Kardia's power to rule, is in peril. The children, joined by their cousin Mary, are called on to stop Mirmah and her chief sorcerer, Archimago. But to do so, they must travel beneath the Northern Mountains, around Goldcoffin's Palace and into the very heart of Mirmah's kingdom. What awaits is much more than a mere adventure.
The villain, Mirmah, is also known as the "Lady of Night." As I recall, she charms Mary with an ice cream sundae the size of a house.
My familiarity with the series ends after this book. From the Web, I glean that some of the later books take place earlier in time, and I have no idea what order it would be best to read them in.