I just remember that the "Hairy" ape like creatures that can use magic get invaded my humans. Supplies run low for the humans so they start to eat the ape like creatures is the one part I distinctively remember.
Your description sounded very familiar. Like you, I read the story so long ago that I couldn't immediately remember the author's name, nor the title. But I could vividly remember the art on the front cover of the paperback edition I had read over a decade ago, and I remembered one or two other details, and so, with a little Googling, I found it. Perhaps this will look familiar:
The Ancient Enemy, by Christopher Rowley, is the first book of a trilogy. (I never got around to reading the later volumes.) The general idea is that Thru Gillo, the hero who is seen on the cover, is a "mot," a race of civilized creatures who seem to physically resemble chimpanzees, and who have been living peacefully for many millennia on a continent which is blissfully free of human influence. There is ancient folklore about "Man the Cruel," but nobody in this part of the world (whichever world it is) claims to have seen a real live Man for approximately one hundred thousand years.
Some members of another local species called "the Assenzi" claim to be old enough to remember Man vividly. The Assenzi also appear to have "magic" or "psychic" powers, although I don't think the regular mots display much talent in that direction. I think many mots feel either that "Man" is just a myth, or, failing that, maybe "Man" used to be a really big deal, but has now been utterly extinct, at least on this world, for so long that it really doesn't matter what "Man" used to be like, back in the bad old days.
This turns out to be a dangerously optimistic assumption! Around the middle of this first book, the mots are astounded when a human fleet sails up to one stretch of coastline on this continent and starts grabbing everything in sight. As I recall, the human soldiers aren't even trying to "conquer" or "enslave" the local mot civilization; they are only trying to butcher the mots and seize their existing towns and assets for the benefit of the human population. The subsequent war is still going strong as this first book concludes. The mots were not diehard pacifists before the invasion-- they already knew about swordplay and archery and other forms of violence -- but they were sufficiently peaceful among themselves that I don't think they had ever developed anything we'd call "a real military organization." However, when faced with strong motivation, they started learning the ropes of warfare, fast!
The humans are all part of a bloodthirsty religious tradition, and, as you recalled, they regard the mots as soulless creatures whom it is perfectly all right to eat for dinner. (To be fair, some of the human characters are handled more sympathetically than others -- they are not wildly enthusiastic about this idea of eating beings who are sentient enough to build houses, weave cloth, keep written records, and learn to speak a human language if given a fair chance.)