Probably Isekai Mahou wa Okureteru!, or in English, The Magic in this Other World is Too Far Behind!
Yakagi Suimei is a modern magician and a high school senior. A genuine scholar of the mystic arts, he follows in the footsteps of his father as he aims for the truth of his world’s secrets. He was on his way home from school with his best friends when a magical circle appeared underneath their feet. On that certain day, Transportation magic!
Suimei was too late in realizing what was happening. He and his two friends, without any way to defend themselves, were sent flying to a parallel world.
…This is the story of the summoned Yuusha’s friend who was also transported into the other world. The story of the friend who journeyed separately from the Yuusha because he rejected the duty of subjugating the mazoku army and Maou. The story of how he learns to live in another world, helps strangers, and winds up alone due to his overwhelming power.
The TV Tropes entry more clearly hits your points:
The kingdom's court magicians perform a summoning ritual for heroes and bring forth a hero and two youths, a boy and a girl. Unlike the girl, the young man refused to fight the Demon King alongside the Hero and demanded to be sent back to his world. The kingdom, angry with his behavior, locked him away.
Now Felmenia stands before the same young man with her strongest magic, the white flame, being completely useless, and asks who he is. It was already obvious to her that this man was far stronger than she was. To this, he simply responds, "Yakagi Suimei, a magician."
- Took a Level in Badass: The magic from the world Suimei and his friends came from outclasses everything the other world has to offer.
- Mage Hunt: Suimei has been hiding his status as a magician out of fear of being hunted down and getting his closest friends in danger.
- Wrong Context Magic: Everyone in the other world mostly use elemental magic, however Suimei uses his own world's magic which is heavily focused on spirituality and religion (i.e. Kabbalah magic) as well as being far superior with the other world's people believing it to be impossible.
Found with a search for isekai "mage summoned", which took me to https://www.reddit.com/r/Isekai/comments/chbvzl/is_there_an_isekai_where/. My rationale for the search terms of that Isekai very seldom have someone magical explicitly summoned, so it'd be a rare combination. I wasn't expecting to get it without the asterisk, though.