Never forget that for every Bran, there are probably thousands of actual raving lunatics.
While the show doesn't particularly draw attention to it, they still hold notions that to us (the modern viewer) sound archaic. One example is the "only men can be knights" scene from the last episode. There are other cases where it's subtly visibly how only a handful of characters (e.g. Tyrion) used "modern" morality that the viewer identifies with, whereas other (even characters we like) still use (to us) outdated ideals.
So look at it with their prejudiced eyes: Bran used to be a healthy boy. Then he became a cripple. Then he seems to have become mentally challenged. He just stares at you with blank eyes, and then sometimes says something that seems to be a non-sequitur.
Tyrion is a longstanding example of how the disabled are treated. The general populace treats them as inferior or avoid them. In Tyrion's case, he has a sharp mind, and more intellectual characters recoginize that Tyrion is not in any way handicapped in conversation (if anything, he outclasses them).
But Bran does not show the same conversational competence that Tyrion does. He speaks weirdly. We, the viewer, know that he's an omniscient entity and therefore know that his utterances are wisdoms. But to someone who doesn't know Bran's backstory, it just sounds like the ramblings of a decaying mind.
In the last episode, he quoted "the things I do for love" to Jaime. To Jaime (and the viewer), that makes a lot of sense. To everyone else, who wasn't there when Jaime said it, it's off-topic and unrelated to what other people are talking about.
This repeatedly happens. Bran references something that makes a lot of sense to someone, but many others who lack context do not understand him and thus assume he is rambling.
Tyrion was the first to recognize Bran's value and that's he's not just rambling, because Tyrion (of all people) knows what it's like to be dismissed as the inferior and broken human. Additionally, Tyrion loves learning, and Bran is effectively an infinite one-man-library.
Bran claims in front of a large audience including Daenerys that the night king has breached the wall and has one of her dragons.
Who knows whether Bran is correct, if Bran is the only one who knows it? Trust is one thing, but confirmation is another. Before it is confirmed, some people may trust Bran, but the vast majority will not believe him yet. By the time it is confirmed, there will already be an air of "Bran says silly things" which is a reputation that's hard to remove.
Not everyone who speaks wisdom (at a time where it is not yet understood to be wisdom) will later be vindicated when it is revealed to be correct.
Similarly, think back to Bran and Littlefinger's conversation, where Bran quotes "chaos is a ladder" to Littlefinger. Since Littlefinger said that to Varys in King's Landing, LF should therefore realize that Bran is omniscient, and must run for the hills because his secrets are known!
Except that he doesn't. LF doesn't understand how Bran knows the quote, but there is a much more sensical assumption: Varys talked about it and Bran found out through some channel. Even if it wasn't Varys, Littlefinger seems likely to assume that someone must have overheard him. There's no reason for him to assume it must be a supernatural phenomenon.
He later speaks in front of the war council in Episode 2, during which many people are present, some who barely know Bran. He speaks of the night king and being the three-eyed raven and the end of the world etc., and no one seems especially concerned.
When someone says something you don't understand, the most common human reaction is to assume that this person is mistaken, rather than thinking you are mistaken.
Because how can you know that Bran isn't just living a childlike daydream in his head? For all they know, Bran is a mentally challenged boy who makes up stories about the things he hears around him.
Never forget that for every Bran, there are probably thousands of actual raving lunatics who are saying nonsensical things.
Think of it this way: your assumption that everyone should listen to Bran because he just might be right; suggests that you think we should listen to every conspiracy theorist because they just might be right.
This is the same mindset that leads people to e.g. retroactively attribute the gift of foresight to Nostradamus, but only after the allegedely foretold thing has happened; which just doesn't make sense.
