This is "A Place Among the Fallen" by British author Adrian Cole. It is the first book of a quartet of books known as the Omaran Saga.
The book was published in 1986 which fits your time scale.
The story starts in dramatic fashion with one of its main protagonists, Korbillian and a companion, being ship wrecked in the world of Orara at a fishing village called Sundhaven. It appears they come just in advance of a tidal wave, but Korbillian is able to split the wave into two which pass either side of the village. Korbillian is described "and the arms that hung at his side were thick, sheathed from elbow to fingers in long black gloves". He wields power by brandishing his gloved hands.
The initial view point character is Brannog although he is a fisherman, not an innkeeper. He has a daughter Sisipher who has some kind of second sight. She sets out with Korbillian and Guile (his original companion). Korbillian says that the world is in mortal danger due to an incursion of alien power through some kind of mound. His mission is to find and destroy this mound and the twisted creature and plants that have grown up around it.
Korbillian has come from another world that was already destroyed in a similar fashion. The hierophants of that world pooled all of their remaining power in his gloves and sent him to Omara to stop it suffering a similar fate. (At least that's what they told Korbillian and he believed them, but they lied to him.)
The story is long and convoluted with a large cast of characters. Although a major character (Simon Wargallow) initially believes that almost no one has innate power and it is his mission to kill anyone (e.g. Korbillian) that does have power it appears that nearly everyone has some kind of magical power in the end.
All of this is just the first novel. In each subsequent novel some of the characters journey somewhere else and find an even bigger threat of ungoverned power. Ultimately it becomes hard to see how anyone could have felt the world was without power, since it seems to be anywhere that anyone looks for it.
Here is the cover of the British edition. It shows Korbillian with power coming from his gloved hands.
