Note: this is a book answer; the film answer might be different.
Short answer
At the time of the Hobbit and LotR, Rivendell is hard to find because it is a small settlement located in a valley in a wilderness region that was depopulated by a war around 1000 years earlier (between Arnor/Arthedain and Angmar).
Longer answer
Under 'Roads' in the index to the Unfinished Tales, we find the following description of the East-West Road:
The Numenorean Road from the Grey Havens to* Rivendell, traversing the
Shire
* My edition has 'of' here, but that's clearly a typo.
So, Rivendell is located on one of the only major roads in Middle-earth. However, the description in the Hobbit shows that the road had deteriorated (not surprising given its age and the aforementioned depopulation).
The only path was marked with white stones, some of which were small
and others were covered with moss or heather.
Note that this comes after the party cross a major ford, presumably over the Bruinen.
Elves from other settlements clearly know the location of Rivendell, as we see from the Council of Elrond, which includes Legolas (from Mirkwood) and Galdor (from the Grey Havens). Also, according to Appendix A of LotR, the Elves of Lorien came to Elrond's aid in TA 1409, so they must have known where to find him. Some of the rangers must know the location of Rivendell, since their leader is fostered there. Dwarves who travel the East-West road might be aware of it; of course after the events of the Hobbit, the remaining members of Thorin's party certainly knew about it, having been shown the location by Gandalf.
After Boromir and Faramir experience the dream about the Sword that was broken, Denethor tells them that
... Imladris [Rivendell] was of old the name among the Elves of a far
northern dale where Elrond the half-Elven dwelt, greatest of lore-masters.
Following this, Boromir set out across Rohan, crossed the Greyflood at Tharbad and headed into Eriador. At this point he clearly knew roughly where he was headed, but at the Council of Elrond, he remarks that he did not know the precise location.
... long have I wandered by roads forgotten, seeking the house of Elrond,
of which many had heard, but few knew where it lay.
In conclusion: those who had been around for a long time (Elves, Istari) knew the location of Rivendell, and those who used the East-West road probably did as well. Others who lived or travelled nearby might well have heard of it, and might also have a vague idea of its location. Elsewhere, highly educated individuals such as Denethor would know of Rivendell's existence and its approximate location, but probably wouldn't know exactly where it was, unless they happened to have been there. A refugee from another part of Middle-earth would probably have no idea about it and would be unlikely to have the supplies to attempt a journey if they did.