Title and author of short SF story
I believe you're looking for R. A. Lafferty's short story "Thus We Frustrate Charlemagne", as suggested in a comment by Eike Pierstorff. The story, which was also the answer to this old question, was first published in Galaxy Magazine, February 1967, which is available at the Internet Archive (click here for download options). It matches your description perfectly.
about a seemingly broken 'improvement' machine
This was Epiktistes the Ktistec machine? Who'd have believed it? The main bulk of Epikt was five floors below them, but he had run an extension of himself up to this little penthouse lounge. All it took was a cable, no more than a yard in diameter, and a functional head set on the end of it.
I think the main point of the experiment is to see if they can change anything:
"We have perfect test conditions," the machine Epikt said as though calling them to order. "We set out basic texts, and we take careful note of the world as it is. If the world changes, then the texts should change here before our eyes. For the test plot, we have taken that portion of our own middle-sized city that can be viewed from this fine vantage point. If the world in its past-present continuity is changed by our meddling, then the face of our city will also change instantly as we watch it.
But they are hoping for improvements:
"We are going to tamper with one small detail in past history and note its effect," Gregory said. "This has never been done before openly. We go back to an era that has been called 'A patch of light in the vast gloom,' the time of Charlemagne. We consider why that light went out and did not kindle others. The world lost four hundred years by that flame expiring when the tinder was apparently ready for it. We go back to that false dawn of Europe and consider where it failed. The year was 778, and the region was Spain.
A small committee sits round a table and invokes the machine.
There are nine of them to start with, counting the machine:
"We have assembled here the finest minds and judgments in the world: eight humans and one Ktistec machine, myself. Remember that there are nine of us. It might be important."
The reader understands that it has changed the whole timeline:
Also one of the characters in the story,
Willy McGilly, a man of unusual parts (the seeing third finger on his left hand he had picked up on one of the planets of Kapteyn's star) and no false modesty
perhaps knows what's going on; at least, he is aware of previous changes in the timeline:
"I hope the Avatar isn't expensive," Willy McGilly said. "When I was a boy we got by with a dart whittled out of slippery elm wood."
"This is no place for humor," Glasser protested. "Who did you, as a boy, ever kill in time, Willy?"
"Lots of them. King Wu of the Manchu, Pope Adrian VII, President Hardy of our own country, King Marcel of Auvergne, the philosopher Gabriel Toeplitz. It's a good thing we got them. They were a bad lot."
"But I never heard of any of them, Willy," Glasser insisted.
"Of course not. We killed them when they were kids."
The number of people at the table is different and people who are still at the table have changed memories. Still those people do not recognise the change
Right. They started with nine (counting the machine), then:
The thirteen of them, the ten humans and the Ktistec, Chresmoeidec, and Proaisthematic machines, turned to the evidence and with mounting disappointment.
And the time after that:
"Did it work, Epikt? Is it done?" Aloysius asked.
"Let's look at the evidence," said Gregory.
The four of them, the three humans and the ghost Epikt who was a kachenko mask with a speaking tube, turned to the evidence with mounting disappointment.
"There is still the stick and the five notches in it," said Gregory. "It was our test stick. Nothing in the world has changed."
"The arts remain as they were," said Aloysius. "Our picture here on the stone on which we have worked for so many seasons is the same as it was. We have painted the bears black, the buffalos red, and the people blue. When we find a way to make another color, we can represent birds also. I had hoped that our experiments might give us that other color. I had even dreamed that birds might appear in the picture on the rock before our very eyes."
and push the button
I think it's a metaphorical button:
"When we, as it were, push the button (give the nod to Epiktistes), this will be changed," Gregory said. "Epikt, by a complex of devices he has assembled, will send an Avatar (partly of mechanical and portly of ghostly construction), and something will have happened to the traitor Gano along about sundown one night on the road to Roncevalles."
[. . . .]
"Push the button, Epikt!" Gregory Smirnov ordered.
From his depths, Epiktistes the Ktistec machine sent out an Avatar, partly of mechanical and partly of ghostly construction. Along about sundown on the road from Pamplona to Roncevalles, on August 14 of the year 778, the traitor Gano was taken up from the road and hanged on a carob tree, the only one in those groves of oak and beech. And all things thereafter were changed.
again and again, each time changing the time line, sometimes to the better, sometimes to the worse ...
They "push the button" three times in the story:
"We will not call the experiment a failure since we have covered only a third of it," said Gregory. "Tomorrow we will make our second attempt on the past. And, if there is a present left to us after that, we will make a third attempt the following day."
I read the story in the 80s, but it might stem from the 50s even.
As noted above, the story was first published in 1967, but it has been reprinted many times (any of these covers look familiar?). In the 1980s, you might have read it in the 1982 Ace edition of Lafferty's collection Nine Hundred Grandmothers.
