The story was about these kids who went down into a hole and ended up in a different place. It started out at night with stars in the sky. I remember there was a mean old man, who was possibly a farmer. It was in my 6th grade English book around 1979.
I'll guess that it's a short novel, it's only one kid, and it's The Forgotten Door by Alexander Key. It has an opening at night (looking at shooting stars) and the mean old farmer. (Also, it was issued by Scholastic, so likely to be encountered in school.)
Do the boy's telepathy, his being from a more peaceful society, a kind family that takes him in, and his wanting to go home sound right?