Agent Smith is the classic character who has been rejected by his peers and masters and seeks absolute revenge.
Defeated by Neo, who he has an inbuilt belief is an inferior being (all humans being regarded as inferior by the AIs), he must avenge that defeat by not merely defeating him in battle, but by defeating his plans.
Smith seeks to take over the Matrix and by doing this presumably would at least challenge the AI outside the Matrix and seeks to defeat the AI in that way (as they need to control the Matrix).
So when Neo allies with the Matrix AI's to defeat Smith, Smith (now a vast self-replicating AI) has an absolute need psychologically to defeat Neo and to humiliate him, and finally to wipe Neo out and replace him with Smith.
Smith is a classic character familiar from all literature. A rejected figure motivated by the need for absolute revenge which requires them to first remove control from their enemies and them to eradicate their enemies in their own way at their own time of choosing.
But I don't quite understand why Smith insisted on engaging Neo in a fist fight, did he expect to kill Neo in that way at all?
No he wants to first defeat Neo, who is acting for the AI as well. Defeat is the first stage of his desired revenge.
Killing is not enough. Your enemy must be defeated in a way that allows you to savor the victory and see they your enemy also understands that they have failed and are powerless.
And that's also a classic type of character for villains.
Why didn't he assimilate Neo when they were struggling ? He seemed satisfied when Neo was down, so what might he do if Neo stayed down ?
Waited. He would have waited until he had extracted the last measure of ego from his victory. Then he would have dispatched Neo with as little thought as you might squash a bug.
Because he seemed to have sensed that something was wrong later, and then assimilated Neo as last resort.
Not a last resort, but part of the need to fulfill a classic revenge fantasy. You first defeat and emasculate your enemy and then, finally, you erase them, make them nothing.
But this was a last resort trap by Neo and the AI, and, arguably, the actually plan was a kind of suicide mission by Neo as a bargain to save the "free" human colony.
This is really nothing more than a classic device in literature to allow the hero to do a grand and noble sacrifice.