It's Sudden, but I Came to Another World! But I Hope to Live Safely seems like a good match.
From Anime-Planet:
"Understanding" is stronger than a sword! There is no magic without weapons! But, this is the story of a man who wishes to live safely! A man stands alone on a mountain full of plants and creatures that he never knew nor saw before. He goes down the mountain by avoiding encounters with bears, slimes and bandits, and is hauled into prison the second he reaches a large city. On top of that, a pretty female knight called Illias manages to break him out and whisks him away to a church. What would a female knight need with a yellow-bellied pacifist? And when will this hair-raising day end already?!
The main character (MC) wakes up in a forest in another world and is attacked by a bear, which is in turn attacked by a slime, inadvertently saving him.
The MC has no apparent powers or weapons, so he sneaks through the forest to avoid being seen until he finally emerges from it and spies a kingdom. However, no one there can understand him, and he ends up being thrown in prison.
He's then visited by a blonde, female knight who puts him in a sack and carries him to a church. At the church, there's a woman who is able to cast a spell that allows them to understand one another.
That's all in the first chapter. In subsequent chapters, the knight does begin to distrust the MC after seeing the way he speaks and behaves in certain situations.
