TLDR: We don't know and can only speculate. The movie uses this to get our curiosity and to establish Thanos as a strong villain.
We never see the Hulk build up anger. For all we know he could have just woken up in his bed, realising the ship is under attack, smashing a few of Thanos underlings and then go right at him, just because he was awoken in the middle of a good dream. At the least, we know he typically gets angrier if he's in life-threatening situations himself or if the love of his life is in danger. In the MCU, Black Widow gets closest to that and she's not around, nor do we have an indication that he was in serious danger before being beaten by Thanos.
So in-universe, it is likely he wasn't at his full potential and not very focused. And Thanos is likely at least about the same strength level, with better technique (less emotional fighting style) in that fight. Hulk was probably taken by surprise in how strong his opponent was - and we've seen in Age of Ultron, that surprising him when he's not looking/focused/at full strength is a sneaky way to get him out of the picture for a while.
Hulk wasn't ultimately defeated in the sense of about to die, yet a space ship is also a terrible place for the MCU Hulk to fight in. So far, we have no indication to think that he would survive in space. Even if that were the case, breaking the ship apart would kill the remaining Asgardians. And this would likely happen if he were to fully "hulk out" and have an extensive fight against someone at least equally strong. So, in his "mentally improved" state where he can talk and recognizes "friends" properly, he might also have held back on purpose. In any case it was probably a sane decision to just send him away and not drag out the lost fight.
Out of universe, the movie explicitly doesn't focus on him or his mental state. What mood he is in, what fights he had earlier whether he just transformed, whether he already is in disagreement with Banner or not. It doesn't make a big fuss about the fight and none about any "preparation" on Hulk's side. This way it builds the question of how he lost and why he hides inside Banner up as a mystery box for the next movie to open and investigate, while for the time being using him and Thor to establish how powerful Thanos is. Up to this point, those two were the top-contenders for "strongest avenger", so it sends a powerful message if Thanos outright beats both(!) - individually.
Personal side note: I sure hope Hulk can pick up that Gimli-Legolas relationship again, solve his mental issues and "level up" as well, after Thor stepped up his game in Ragnarok and IW.