This may be Special Deliverance, by Clifford D Simak. Published in 1982.
From the Amazon link:
It all started when Professor Edward Lansing wanted to know who really wrote that great term paper on Shakespeare and learned that his student had bought it from a slot machine. Going to investigate, the good professor found the machine, which gave him two keys and sent him in search of other slot machines. The third machine he tried took his money and transported him to a strange new world. Here Lansing meets up with an odd assortment of fellow travelers--including a take-charge Brigadier, a pompous Parson, a female engineer, a lady poet, and Jurgens, a caretaker robot--all of whom are as mystified as he.
Here's the card players:
Chapter 22 (near the end):
Lansing finished his drink and set the mug upon the table. He got slowly to his feet and walked across the room until he stood beside the table where the four were playing cards. He stood for a long moment, with none of them paying him the slightest heed, as if they had not noticed his approach. Then one of them raised his head and turned it, looking at him.
Lansing stepped back a pace, horrified at what he saw. The eyes were dark holes in the skull, out of which peered two black obsidian pebbles. The nose was not a nose, but two breathing slots, slashed into the area between the eyes and mouth. The mouth was another slash, without benefit of lips. There was no chin; the face sloped down to the neck in a slanting line.
The card players keep turning up whenever the travellers stop at an inn.
