Insidekick by J. F. Bone
https://www.isfdb.org/cgi-bin/title.cgi?41279
The man is about to be executed or is wanted by the police
Satisfied
that it was empty except
for Fred Kemmer and himself, he
sidled up to the Earthman’s desk
and hissed conspiratorially in his
ear, “Sir, this Johnson is a spy!
Is it permitted to slay him?”
“It is permitted,” Kemmer said
in a tone suitable to the gravity
of the occasion.
He's infected by a parasite/symbiant
CURIOUSLY, the Zark sent out a pseudopod of its substance
through the open mouth of its disguise.
The faintly glittering thread
oozed downward and struck Albert’s
head beside his right eye.
Without pausing, the thread sank
through skin and connective tissue,
circled the eyeball and located
the optic nerve. It raced inward
along the nerve trunk, split at the
optic chiasma, and entered the
corpora quadrigemina where it
branched into innumerable microscopic
filaments that followed the
main neural paths of the man’s
brain, probing the major areas of
thought and reflex.
The Zark quivered with pleasure.
The creature was beautifully
complex, and, more important, untenanted.
He would make an interesting
host.
The parasite notes that it's nice to see in colors.
Color was something new to
the Zark. Its previous hosts had
been color blind, and the symbiont
wallowed in an orgy of bright sensation.
The man escapes from pursuers by teleporting
He suddenly felt alone and helpless,
wishing desperately for a quiet
place where he could dress his
wound and be safe from the eyes
he knew were inspecting him. He
was too conspicuous. The pajamas
were out of place on the street. Undoubtedly
natives were hurrying to
report him to the IC.
His mind turned to his room
in the hostel with its well-fitted
wardrobe and its first-aid kit —
and again came that instant of utter
darkness — and then he was
standing in the middle of his room
facing the wardrobe that held his
clothing.
HE felt no surprise this time.
He knew what had happened.
Something within his body was
acting like a tiny Distorter, transporting
him through hyperspace in
the same manner that a starship’s
engine room warped it through the
folds of the normal space-time continuum.
There was nothing really
strange about it It was a power
which he should have — which any
normal man should have. The fact
that he didn’t have it before was
of no consequence, and the fact
that other men didn’t have it now
merely made them abnormal.
The story can be found in the Internet Archive
https://archive.org/details/Galaxy_v17n03_1959-02