He shoots some kind of Palpatine-affiliated guard in the shot before he encounters the Knights of Ren (sorry for the blurry image):
Since the Knights of Ren are apparently working with Palpatine at this point (for unexplained reasons; possibly because the rest of the First Order has been subsumed by the Final Order), they probably saw or heard this happening and correctly surmised that Ben had gone rogue.
Additionally, the Knights of Ren were Force-sensitive to some degree, according to the Visual Dictionary. When Kylo Ren approaches them on Exegol, he's (apparently) already turned back to the light, or at least doesn't seem to still be consumed by the dark side. Since they had spent a good amount of time with him, given that he was their leader, they may have sensed that something was wrong.