I wonder if the location of Rivendell is a secret known only by the wise such as Gandalf, or is it public to almost everyone in Middle-earth?

As seen in the movie, Gandalf leads Blibo Baggins and the Dwarves through a narrow tunnel to escape from Orcs, none of them reconigzed that they were on the way to Rivendell. Also we see that Rivendell is not flooded with immigrants and poor people seeking refugee (compared to the Medieval Europe in our world, the life in Rivendell should be rather attractive for most people, except the richest one; If I live in Middle-earth, the first thing I would do is immigrate to Rivendell), but only a few Elves live a secluded life. Does this suggest that the location of Rivendell is a secret?