Many should recognize that experience. It doesn't come with guarantees. Hearing the whispering of the Spirit doesn't mean that a person doesn't have to figure out what it means, nor does it mean that we can't be mistaken as we try to figure that out. But in spite of those caveats, the experience is real. Many religious people, LDS or not, can testify of that.
Sometimes it may sound like Mormons ascribe something almost like infallibility to their prophets, but Mormon prophets have not claimed infallibility. Faced with saints who believed that a prophet is always a prophet, Smith said, "a prophet was a prophet only when he was acting as such" (History of the Church 5:265). That is a quotation often repeated by Mormons when they argue about Mormon belief, a reminder that we can't establish doctrine simply by referring to something once said by a Mormon authority.
Mormon prophets, like those of the Bible and Book of Mormon, are products of the time, place, and culture in which they live. But their message is no less prophetic because of that. In fact, it is impossible to think that they could have any message at all if they were not, to an important degree, products of their age.
As one of Smith's early revelations (1831) says, God's revelations are "given unto my servants in their weakness, after the manner of their language, that they might come to understanding" (D&C 1:24). So no prophecy can be understood apart from the context (the human weakness and language) in which it occurs: because of the time and culture in which prophets live, they can make mistakes that are products of that time and culture.
But time and culture are not the only explanations for the possibility of prophetic error. Prophets can also be mistaken because they are human beings. Speaking of himself Smith said "the flesh is weak subject to like passions with [i.e., like] other men" (Ehat and Cook, eds., The Words of Joseph Smith 233). As the scriptures show over and over, God calls people to be prophets who are fully human.
But if prophets can be mistaken, then what distinguishes them from other human beings? Is it perhaps their piety? Not according to Smith, who said (according to the journal of an early Mormon convert):
Brethren I am not a very Pieus man. I do not wish to be a great deal better than any body else. If a Prophet was so much better that any body else was he would inherit a glory far beyond what any one else would inherit and behold he would be alone, for who would be his company in heaven. for If I should condescend to be so righteous as the brethren would wish me to be, lo I should be taken from your midst and be translated as was Elijah. Righteousness is not that which men esteem holiness. That which the world call righteousness I have not any regard for. To be righteous is to be just and merciful. If a man fails in kindness justice and mercy he will be damed for many will say in that day Lord, have we not prophecie in thy name and in thy name done many wonderful works but he will say unto them ye workers of iniquity &c (Martha Jane Knowlton Coray notebook; 21 May 1843; original spelling, punctuation, and grammar retained)
The righteousness of a prophet is to be decided in the same way it is decided for other people, on the basis of his justice and mercy rather than on the basis of his position in the Church or society. I assume that LDS prophets are righteous people, but they are righteous because they are just and merciful not because they are prophets.
What, then, sets the prophet apart from other persons if he can be mistaken and he is not necessarily more righteous than others? That, in virtue of his calling, he has the solemn authority and heavy responsibility to receive the word of God for the body of the Church and to preach the gospel to the world. The prophet of the LDS Church is a human vessel through which divine inspiration comes. Mormons throughout the world love, respect, and honor him for that, not because he is smarter or holier or more important than anyone else.