Book written before 2000. My recollection is exceedingly vague, but what I recall is an individual, or a group of individuals, moving to a house that is either next to or in the outskirts of a large forest. At some point, early in the story, one of the inhabitants sees a strange man coming into a yard (or courtyard, or possibly even a clearing in the forest visible from the house) who, it turns out, is not actually real.
If I recall, the significance and impact of the phantasmal people escalates as the story goes on; I also dimly recall at least one character, in an effort to resolve the situation (or, at least, figure out what is going on) tries to penetrate to the center of the forest, which may house the cause or source of the apparitions.
The people are physically tangible, I believe - they can touch things and interact with other people, but I believe they are prone to disappearing (and, possibly reappearing) and various other anomalous behaviour.