"Thus I Refute Beelzy", a short story by John Collier, published in the October 1940 issue of The Atlantic Monthly. It was reprinted in Famous Fantastic Mysteries, October 1952, which is available at the Internet Archive. A reading by Nelson Olmsted from a 1957 episode of the radio program Sleep No More is also available at the Internet Archive. Here is a plot summary from Plot Spot:
A little boy, Small Simon, refuses to acknowledge to his father, Big Simon, that Mr Beelzy is nothing more than a daydream. The father attempts to reason with his son, laying forth his arguments in a logical and straight-forward manner, but then resorts to threats of violence when his son fails to heed him. Small Simon seems unconcerned—he claims that Mr Beelzy won't let anyone hurt him...
There is an anthology I had as a kid. The first story
Hmm. The only book I've been able to find with "Thus I Refute Beelzy" as the first story is the 1974 trade paperback anthology Fantasy: Shapes of Things Unknown, but the cover is nothing like your description. Is any of these covers the one you remember? If you read it in a Bantam paperback, it might have been a Bantam edition of Collier's collection Fancies and Goodnights.
was about a child who had a monster/creature, maybe named Mr. (something or other).
"He is six," said her husband. "He is a reasonable being. He must choose for himself. But what game is this, Small Simon, that is worth getting nervy and dazed over? There are very few games as good as all that."
"It's nothing," said the boy.
"Oh, come," said his father. "We are friends, are we not? You can tell me. I was a Small Simon once, just like you, and played the same games you play. Of course there were no aeroplanes in those days. With whom do you play this fine game? Come on, we must all answer civil questions, or the world would never go round. With whom do you play?
"Mr. Beelzy," said the boy, unable to resist.
"Mr. Beelzy?" said his father, raising his eyebrows inquiringly at his wife.
"It's a game he makes up," said she.
"Not makes up!" cried the boy. "Fool!"
"That is telling stories," said his mother. "And rude as well. We had better talk of something different."
The father did not believe the creature existed.
"I hope you are listening to me," said his father. "All you have to do is say, 'I have been playing a game of let's-pretend. With someone I make up, called Mr. Beelzy.' Then no one will say you tell lies, and you will know the difference between dreams and reality. Mr. Beelzy is a day dream."
The story ends with the family finding the father's hand on the landing of the staircase.
Foot, not hand:
It was on the second-floor landing that they found the shoe, with the man's foot still in it, like that last morsel of a mouse which sometimes falls unnoticed from the side of the jaws of the cat.