It was in an anthology, the kind of little book (pocket-sized, not magazine sized) you used to find in the checkout lanes in grocery stores. I think Frederick Pohl's story "Spending a day at the Lottery Fair" was also in that book, but it might not be. Trust me, I have scanned any list of a Pohl anthology to no avail!
What I can recall is that the protagonist was an adult recalling a childhood memory of an odd reclusive man who lived on his street when he was a kid. The man's appearance (dark glasses, frail, blue-ish lips) had adults thinking he had a heart condition but the kids just knew there was something not right with the man. The final lines are the now-adult recalling a conversation wherein the man removed his dark glasses and squinted his eyes shut and said "I will never get used to the brightness of your sun".
Any help would be appreciated.