The story, as far as I can remember, played in a manor/land house. A boy, living there with relatives, (or maybe just visiting, but without his parents), one day sees strange children in the garden, which turn out to be ghosts of former residents, many years ago. His relatives/hosts obviously know about the ghosts, and tell him stories about the time these children lived there. While the stories told contain some tragedies, it was not a spooky book.
From the context I'd guess it was targeted at children in the 10-12 years age group. I read the book in German, but I guess it must have been translated from English, as the whole setting seems very British to me. I think I read the book sometimes in the 70s, but I don't know when it was written. I seem to remember that there was more than one book, but I'm not sure.