Because the Castle was cursed and because the Beast has never visited the village.
More specifically, the castle is (or at least was, in an earlier version of the script) surrounded by a magical mist that cloaked its presence from the villagers both physically as well as resulting in them not remembering its presence next to the village. It's worth noting that the Beast hasn't left the castle since his punishment.
"Since you are no better than a beast, than you deserve to look like one
...
And you will remain a prisoner in this enchanted place with no human company."
...
'This enchanted Mirror will show you any pan of the wide world you wish to see. Look well...for it's a world you can no longer be part of."
...
The beast/child races desperately through the castle...up to the
highest tower. He looks down to see her disappear into a deep,
enchanted mist which has surrounded the castle. "I'm sorry," he cries.
"Please...come back! I'm sorry!" And as the beast/child peers
desperately out through the bars, we pull back to:
The enchanted mist can be breached by accident, but only through a woodland guarded by fearsome magical wolves.
Maurice turns around and looks up. He gasps! For there....looming
ominously over him....cloaked in mist...a huge dark, forboding castle!
It glowers, lonely and sullen as if it's been brooding all these years
over its solitude
The castle can be reached by using the magical mirror as a guide.
TEAPOT: I wouldn't trouble you ..but the castle is under attack.
WARDROBE: Oh dear, this is most disagreeable! However did they find us through the mist?
TEAPOT: They have the Mirror.
When the spell is lifted, the cursed mist lifts and the villagers are able to see the castle once again.
Dawn is breaking as the castle casts off the spell that has held it
captive for so long. The veil of mist lifts from the surrounding
forest. The sun sparkles on the morning dew. Birds chirp. And the
castle itself seems to shimmer.
In the new 2017 adaptation (or rather the official novelisation of the new 2017 adaptation) it's made even more explicit what's been going on.
“Oh, Chip!” Mrs. Potts cried happily. “Look at you—you’re a little boy
again!” Reaching out, she tried to give his cheek a squeeze. He ducked
out of the way, like any little human boy would, and raced toward the
front door. As he flung it open, the sun poured in—and so did some of
the villagers.
For truth be told, they had been under the enchantment, too. Now, with
every moment that passed, they were beginning to remember all they had
forgotten: the castle with the cruel king and the haughty prince, the
lavish parties that had once been thrown, their loved ones who had
worked there.