Is there any evidence of a force user using a kyber crystal that is wrong for them in their lightsaber and this having an effect on their ability to use it? How does it impact them?
Not in any significant way. In fact, it's just the opposite. In current canon, the crystal is affected by the user.
In the novel Ahsoka, she takes the kyber crystals from an Inquisitor's lightsaber and puts them in her hilts. They then changed color in response to her attunement.
Before she could even consider shouting a warning, the red lightsaber burst into a mess of noise and light. Bright spots pricked at her eyes, and then all was quiet. The creature wasn’t going to bother her anymore.
Kneeling beside her fallen foe, Ahsoka sifted through the wreckage of his lightsaber hilt. The crystals that had powered his lightsaber were no longer contained by metal, but their song had not dimmed. She held them in one hand, almost shaking as the familiarity of them coursed through her, while the other hand retrieved the half-finished hilts she carried with her.
Her mind’s eye sorted the preassembled components and those she had just retrieved into order, locking each into place with the others. When Ahsoka opened her hands, she was not surprised to find that two lightsabers, rough and unfinished, were waiting.
They would need more work, but they were hers.
When she turned them on, they shone the brightest white.
Ahsoka - Chapter 28
The novel hints at the crystals being Ahsoka's which shows that even though they were wrong for the Inquisitor, it did not have a noticeable effect. And once Ahsoka got them, she was able to "reattune" them.
The Star Wars Databank says:
A lightsaber crystal is colorless until first attuned and connected to a Jedi -- at which times it glows either blue or green or, in some rare instances, another shade. From that point on, it retains that hue.
Kyber Crystal | Star Wars Databank
So all evidence points to the crystal being the only one affected. Now, since in current canon a crystal is attuned with the Force and an extension of a Force-sensitive's awareness, it is likely that using a crystal that is not yours (or using someone else's lightsaber) has some disadvantages. But this would be much like using any weapon that was not the one you were used to using.
In summary: The crystal will work and there will be little to no effect
If the crystal is not attuned to user, he could not use it to full effect
First example is from comic Darth Vader: Dark Lord of the Sith part 1. I could not copy all relevant pages here, but basically Palpatine tells Vader that his old blue saber is no longer good enough for him because it was Jedi weapon. Instead he needs to bleed Kyber crystal and build new one . It is implied that he lost his fight vs Kenobi because he was no longer attuned to the weapon.
Second example could be Luke Skywalker. He firstly used that same blue lightsaber (belonging to Anakin) but he was not very proficient with it, losing fight and hand vs Vader. Later, he constructed his own lightsaber (green) with much better results (beating both Vader and Jabba's henchmen)