If we treated all ideas by their worst defenders, none would survive.
Part of the criticism that libertarianism is blind to non-governmental power and coercion is on-point. It does seem to be a weakness of the ideology generally. If you dig into the weeds of the thing, though, it is not a point that remains unadressed. Libertarian thought traces the coercive power of corporations, for example, to their ability to rent-seek, that is, to convince government and other power brokers to form regulations that privilege them, hurt their competitors, reduce or eliminate methods to recapture externality losses, and generally distort the market. Their point, shortly, is that such power is really a consequence of their success at co-opting government power. It's not an argument that's lightly dismissed, and economists of all ideological stripes take it seriously.
...or say that the difference is that government has an exclusive monopoly on the use of force.
Which is the entirely uncontroversial Weberian definition of the state (technically, you need to throw a 'legitimate' before 'use' to match the definition exactly, but that turns out to be trivia because government power is self-legitimating by definition).
In the last case, they tend to wind up sounding like they think that taxes are collected literally at gun point.
Try not to pay your taxes and see what happens. Push it far enough, and eventually men with guns will come to take you away. It's just that there are a few layers of refusal generally standing between the average person and that end. So it is not at all inaccurate to say that compliance with, say, taxes, is every bit guaranteed by force. This is basically true of any government law or regulation; refusal to obey may, depending on how serious the rule is, immediately cause force to be brought to bear, or it may take a few more obstinate steps to make manifest, but the chain of action always ends with men with guns as the guarantor of compliance.
Which makes this...
"I have no idea why they feel that way..."
...problematic. You do know, because they told you that they distrust the government to wield their monopoly on the legitimate use of force, as you yourself admitted. You can *disagree* with that, but that's not the same thing as what you're doing here, which is a none-too-pleasant rhetorical trick.
Pretending your opponents are idiots who don't know why they believe what they believe is akin to Christians telling atheists they really believe in God but are just in rebellion or some-such. If I judged liberalism by every stupid liberal who didn't know how to defend his or her own ideology, I would disclaim the whole ideology as bankrupt too. Rather, seek out those who can contextualize and make detailed arguments in favor of their ideas.
If I could get everyone to read two books, it would be John Rawls' "A Theory of Justice" (the foundation of modern liberalism) and Robert Nozick's "Anarchy, State, and Utopia" (the foundation of modern libertarianism). It's not like these ideas haven't been thought through, or have no heft. It turns out that there are very good arguments to be had on every side, which people dismiss at their own intellectual peril.
