The Mabus Problem: Fighting Anonymous Harassment on Social Media

The Mabus Problem: Fighting Anonymous Harassment on Social Media September 8, 2014

Gargoyle

I’m not much of a video gamer (I spend my free time writing!), but I’ve been following the work of Anita Sarkeesian, a media critic whose non-profit group Feminist Frequency critiques sexism in popular culture, with a special focus on video games. Her series Tropes vs. Women in Video Games points out lazy, sexist clichés that game designers resort to repeatedly: damsels in distress, women in refrigerators, scenes of sexual violence just to make the game world darker and “grittier”, and so on.

But what’s astonishing is the onslaught of hatred and violent threats that’s been directed against Sarkeesian by knuckle-dragging misogynists in response to what is, honestly, a fairly mild critique. When she was raising money for her series, one angry sexist made a video game where players could (virtually) punch an image of her face. Most recently, she’s received death threats so violent and specific, she had to temporarily leave her own home.

Whether they know it or not, the trolls and misogynists who engage in these tactics have already lost the debate. By resorting to crude harassment and thuggish threats, just as religious fundamentalists do, they’re admitting that they have no valid argument – that they can’t rationally defend their regressive beliefs. Like all kinds of fanatics, the only thing they can do is to raise the cost of critiquing them by launching a barrage of verbal and emotional abuse at people who point out the ways they’re wrong, trying to silence their critics through fear rather than reason. But they try to make up for their intellectual vapidity with sheer volume, using the same harassing tactics on platform after platform.

Jezebel, a feminist news and opinion site that’s part of Gawker Media, was besieged by persistent trolls using anonymous accounts to post violent pornography in its comment sections. Despite repeated complaints from Jezebel’s writers (who also have to moderate comments on their posts), Gawker’s management ignored the problem until the writers united to pen a joint open letter of protest, which finally shamed them into introducing some fixes.

Twitter is another site with a chronic harassment problem, exacerbated by its open design that allows anyone to tweet at anyone else without restriction. This means that most prominent Twitter users, especially women, face a constant trickle (sometimes rising to a flood) of rape threats, violent misogyny and other abuse from no-name trolls. The only tools Twitter has offered are the ability to block and report harassing accounts, but both options are utterly useless since, again, harassers can create new accounts and be back within minutes. The single-minded obsession of the worst actors is well over the line into sociopathy: for example, the feminist writer Imani Gandy has written about one hateful individual who, has been creating multiple throwaway accounts every day for two years just to hurl racist and misogynist abuse at her.

Given its history of welcoming some of the worst people on the internet, it’s no surprise that Reddit, too, is a hive for this problem, as in the recent coordinated troll attacks on a subreddit for black women’s issues and another for rape survivors. Again, Reddit’s design aids and abets the bad actors, as moderators have no option but to individually ban accounts posting this garbage, which does nothing to stop the same individuals from creating new accounts and coming right back to resume their harassment.

I suggest that this should be dubbed the Mabus Problem, after a particularly infamous and persistent user of this tactic who waged a one-man campaign of harassment and death threats against prominent atheists and skeptics for years, using literally thousands of throwaway e-mail, bulletin board and social media accounts. He was finally shut down after a massive protest campaign prodded the police to arrest him, but there’s no way this solution can scale to encompass the volume of anonymous harassment that happens every day on the internet. (As many targets of harassment can attest, getting the police to act is a Sisyphean struggle, even in cases of clearly criminal behavior.)

What’s so frustrating is that, at least from a technological standpoint, this is a fixable problem. Trolling is really just another form of spam, and I can’t remember the last time a spam e-mail leaked through into my Gmail inbox, for example. There’s absolutely no reason, other than inertia and indifference to the problem, that social media sites can’t put similar safeguards in place. Of course, most technology companies are owned and staffed by white men who simply don’t experience online harassment to anywhere near the degree that women and minorities routinely do, so they have little incentive to care about it. Often, they’re oblivious even to the existence of the problem.

Stepping into the gap where social media companies have failed to act, there are third-party projects like Block Bot and Block Together, which allow blacklists to be shared by many users, but this is at best a partial solution. A truly comprehensive solution would require changes by the social media companies themselves. I can think of several simple and effective countermeasures against trollish harassment:

Encourage user accounts to accumulate history. Sites like Twitter, for example, could have a “filter out all tweets from accounts less than X days old” setting. Of course, trolls could still create and “age” throwaway accounts, but it would deprive them of immediate satisfaction, which I’m willing to bet would be a major disincentive to the typical immature internet misogynist. Alternatively, there could be a built-in waiting period, from several hours to several days, before a newly created account could be used. Again, the idea is to deprive trolls of the immediate gratification of getting to resume their harassment instantly when they’re blocked.

Block by IP address. A solution so simple it’s incredible more social media sites haven’t implemented it: allow a user to block not only a given account, but any other account with the same IP address. Since these addresses are relatively static, it would be harder and harder to usefully create new throwaway accounts for harassment.

Banned words lists for auto-blocking. Rather than manual, reactive blocking, which requires targets of harassment and abuse to absorb it all, why not make proactive blocking possible? Allow a user to define a list of banned words; anyone who directs a post at you using one of those words is automatically blocked. It wouldn’t be difficult to imagine what sorts of words and phrases could go on these lists.

Finally, the most effective solution would be true Bayesian filtering. This is what Gmail uses: algorithms that “learn” what’s bad by analyzing comments flagged as offensive and building up a statistical model of terms that most frequently appear in them. If harassers shift their tactics, the filter shifts with them.

If social media companies truly made it a priority to stop abuse and harassment of their users, they could put any of these measures in place in short order. What’s keeping them from doing it?

"I'm sorry it seems that way."

The Fountainhead: Jury Nullification
"why did you ask how it was knownWell yeah, that's my reaction any time someone ..."

America’s Functional Atheism
"These propositions aren't contradictory. We've seen that Christian fundamentalists vote consistently, while the nonreligious don't. ..."

America’s Functional Atheism
"Of course religious believers have doubts. How could they not?So, if this is an "of ..."

America’s Functional Atheism

Browse Our Archives

Follow Us!


TRENDING AT PATHEOS Nonreligious
What Are Your Thoughts?leave a comment
  • crashfr0g

    Your technical analysis leaves something to be desired, here – trolling isn’t a form of spam, in fact it’s basically the opposite of spam. Whereas a spam message is often auto-generated to appeal to as many people as possible, and broadcasted to as many email addresses as possible, trolling and harassing messages are usually singletons, hand-crafted to shock, intimidate, and assault an audience of one.

    The problem is unsolved because it is hard to solve – bayesian modeling, IP blocking, and banned word lists either don’t work or block far more legitimate messages than harassing ones. Require Twitter users to post from “aged” accounts and they’ll just set up a system to pre-age accounts.

    I think you’re being naive here. While I don’t doubt there’s a certain lack of immediacy resulting from the demographics of the average computer programmer, there’s no shortage of female programmers – top-flight talent who themselves have directly experienced sexist harassment on the internet (I’m thinking of Kathy Sierra, for instance) and who would be highly motivated to solve the problem, and capable of solving it, if indeed the solutions were as easily reachable as you suggest.

    But it still isn’t solved. I don’t think there’s going to be a technomagic solution, because if there was, we’d already have it.

  • James Picone

    I am a programmer, long-time reader of your blog, and agree that the harassment of people like Anita Sarkeesian is utterly ridiculous and the opinions of her harassers are wrong. I also think your analysis is deeply, deeply flawed here. I’ll go in depth.

    The difference between something like the Bayesian filters used for email and anything you could apply for something like Reddit is that email is a one-to-one communication service and Reddit is intending to model a forum where there’s many-to-many communication. User A sets up filtering that happens to detect a comment by user B and blocks it. What do you do with comments following on from B’s post? Not show them I guess, because otherwise they don’t make any sense. Hopefully that doesn’t leave out anything useful.

    Useful emails tend to be noticeably longer than spam. That’s not the case with Reddit, which makes the filtering problem harder. If an email to you talks about ‘rape culture’, it likely has enough volume for Bayesian analysis to weed it out. If a reddit commenter says “I’d like to rape your culture” that’s not much different from “This is an example of rape culture” from a filtering point of view, and obviously the first comment should be blocked and the second shouldn’t. This problem is even worse with Twitter.

    Bayesian filtering has a much higher false positive rate than you know, I suspect. Try going through your Gmail spam box at some point and count how many messages aren’t actually spam. Given the actual volume of spam-to-real-message, it’s actually surprisingly large. The classic problem for this kind of filtering are things like websites for breast cancer, where the context makes the word ‘breast’ much less likely to indicate content that the filter is intended to block. Solving that in the general case requires solving the strong AI problem – every other fix is just a heuristic around the edges.

    IP blocking doesn’t make any sense with respect Twitter or Reddit. Accounts don’t have an IP, computers do, and those are not the same thing. Account A is logged on from IP B, which is blocked, and tweets. Block the tweet. What if account A is then logged on to from IP C (say a computer at a library) and tweets. Block it, don’t block it? The former means you have to keep a pool of every IP ever used for an account and block it if any one of those IPs is in your blocked pool, which magnifies the issues with using IPs to try and identify a user (they do get reassigned more than you think. Come next month that IP has been reused for a different customer of the ISP. Suddenly you can’t see their tweets. Whoops). The latter means you can dodge the IP block just by using the internet somewhere else or by power-cycling your router or using a proxy.

    Filtering accounts less than age X functions, but is absolutely toxic to the business model of places like reddit and twitter. So you’ve just made an account and they want to encourage you to use their site, but because you’re new your account is in the new-person-ghetto and either you can’t do anything or a substantial fraction of the userbase have you blocked, so the site isn’t useful to you. You can see why websites wouldn’t want that, right?

    Banned word lists are just a special case of Bayesian filters – all the same problems apply.

    I’m not sure if there’s a technical solution to this problem. The closest that comes to mind is applying the Reddit upvote/downvote mechanic to people. Either match your upvote/downvote tendencies with other user’s, use their upvotes/downvotes proportional to how much you agree with them to help weight your posts, give very low weight to users with little to no voting background (i.e. new users). Still has the problem of putting new users in the ghetto, but resolves itself quickly if they produce content people in your circle often upvote. But it does have the problem of putting people in their own bubble, where they only see stuff that other people who think like them have upvoted.

  • unbound55

    I do think your post is well thought out and generally very accurate. However, I’m compelled to pick a few nits (it’s me, not you). Former programmer here that moved to management a long time ago…so not current with the latest techniques, but aware of some of the interesting programming possibilities.

    I disagree that Bayesian filters can’t be used for places like Reddit from a technology perspective. It is, to one of your other points, completely against the business model, and I suspect that makes the application of Bayesian filters a non-starter more than the programming hurdles involved. I’ve seen some interesting application of different types of filters over the years, so I think they can be applied to this situation (albeit with major effort).

    The other nit is how effective the spam filter at Gmail is. Keep in mind that for the filtering process to be effective, you do have to take some time to review the spam folder in the first few months and let it know what is actually a false-positive. That is an important thing to understand for any of the advanced filters…you do have to take the time to finish off the training of the filter. For my primary account, I took the time to essentially unflag what was actually good e-mails in the first few months. For a very long time now, the number of false-positives is so rare that they are memorable when they happen…in fact, despite weekly reviews of my spam folder (I can’t afford to have missed mails on that account), I’m struggling to recall the last false-positive e-mail that fell into the spam folder (more than 4 months ago to be sure). I average about 200 spam messages a week, so over a period of 16 weeks, one false-positive is really an amazing rate (about 0.03%).

    Overall, I agree that the technical solution would be very challenging in a forum environment, but I disagree that there can’t be a technical solution to help reduce the issue to be a bit more manageable. It would take a very serious effort though, and, short of a lawsuit forcing the issue, I don’t see the owners of the websites willing to take on that large of an effort.

  • James Picone

    I guess the other problem with Bayes here is that the assumption with spam filtering is that most mail is spam and very little is ham. For filtering out abusive messages, that’s not globally true – Zoe Quinn or whoever probably gets more abusive messages than useful ones, but most users get more useful than abusive. I guess you make the filter an option and leave it off by default, though.

    I’m really just not sure there’s enough room to distinguish abusive and useful messages in Twitter. Might be room in Reddit. Honestly i should probably just test this instead of making an arse of myself here. I don’t see ‘rape’ or ‘feminist’ being useful classifiers, although Markov-style techniques probably distinguish ‘rape culture’ and abusive use. ‘SJW’ and variants are clear, but then they just avoid using them. Not sure about swearwords or terms like ‘crazy’.

    It’s an interesting thought experiment, though. Lots of subtleties. You can’t let accounts know that the Bayesian filter has blocked one of their messages, or otherwise they can just train around the filter. You’d have to block bare links because otherwise they’ll just link to obscene pictures or text or whatever. Not sure how you’d handle faux-useful tweets with links to something unpleasant.

    Silent blocking would probably be a useful feature in general – being able to block twitter users without making it clear to the blocked user that they’re blocked. Messages from Eve to Alice just don’t get shown to Alice. That might already be how it works, I don’t know, I don’t use Twitter much and haven’t ever needed the admin tools, I don’t use Reddit either.

    The only good solutions we have for these kinds of problems in other arenas rely on features that aren’t present for websites – people making new phone accounts just to call someone to harass them get shut down by the police because there’s identifying information associated with the account, people doing it in meatspace similarly run into the police. I guess requiring some identifying information before being able to create an account works, or some kind of two-level system where there are anonymous accounts and identified accounts and you can choose only to see identified accounts, but I’m not sure giving up anonymity is a great idea here.

  • Azkyroth

    But it still isn’t solved. I don’t think there’s going to be a technomagic solution, because if there was, we’d already have it.

    And someday there will be a world market for maybe six computers.

    Except for the details of Bayesian filtering, perhaps, the solutions Adam outlines already exist, except for the final code implementation in context. The problem is that the people who control the back-ends of the social media platforms in question are not motivated to implement them*, as stated with examples in the post.

    Try to keep up.

    *A problem likely to be exacerbated by smug assurances that the problem can’t be solved, by people who don’t appear to be affected by it. *cough*

  • Whereas a spam message is often auto-generated to appeal to as many people as possible, and broadcasted to as many email addresses as possible, trolling and harassing messages are usually singletons, hand-crafted to shock, intimidate, and assault an audience of one.

    That’s not as much of a difference as you think. Spammers also try to personalize their messages as much as possible, by adding randomized text and such, because a mass broadcast is easier for filters to flag.

    The problem is unsolved because it is hard to solve – bayesian modeling, IP blocking, and banned word lists either don’t work or block far more legitimate messages than harassing ones.

    Based on what evidence? Which social media sites have tried these measures and found that they didn’t work?

    Require Twitter users to post from “aged” accounts and they’ll just set up a system to pre-age accounts.

    That’s like saying that if we put up anti-suicide nets on the Golden Gate Bridge, people will just go to the next bridge over and jump from there instead. In fact, that’s not how it works: suicide is a highly impulsive and spur-of-the-moment decision, and people who find their first choice of means blocked often won’t try another. I’m predicting that social media harassment works the same way, and while people could in theory invest elaborate preparation into getting around these measures, the vast majority of them won’t.

    there’s no shortage of female programmers – top-flight talent who themselves have directly experienced sexist harassment on the internet (I’m thinking of Kathy Sierra, for instance) and who would be highly motivated to solve the problem, and capable of solving it, if indeed the solutions were as easily reachable as you suggest.

    Kathy Sierra’s programming skill is irrelevant unless she worked at, and more importantly had policy-making authority at, the social media companies through which she was receiving the most severe harassment. And as I said, part of the problem is that these companies are institutionally dominated by people who rarely get harassed and have little incentive to see the problem solved.

  • User A sets up filtering that happens to detect a comment by user B and blocks it. What do you do with comments following on from B’s post? Not show them I guess, because otherwise they don’t make any sense. Hopefully that doesn’t leave out anything useful.

    Well, in Reddit’s case, the filtering could be at the subreddit level and would block the comment entirely (or put it into a moderation queue), rather than screening it out for only a specific user.

    The classic problem for this kind of filtering are things like websites for breast cancer, where the context makes the word ‘breast’ much less likely to indicate content that the filter is intended to block. Solving that in the general case requires solving the strong AI problem – every other fix is just a heuristic around the edges.

    I’d be willing to bet that most women targeted for harassment would be very glad to accept the tradeoff of blocking an occasional legitimate message if it meant that abuse could be reliably screened out.

    The latter means you can dodge the IP block just by using the internet somewhere else or by power-cycling your router or using a proxy.

    Yes, I’d agree that this option could result in many public IPs (libraries, coffee shops) eventually being screened out. Again, I imagine this is a tradeoff that most victims of harassment would be quite willing to make. Nor is this a new issue: many websites that accept comments have an option to block open proxies for precisely this reason.

  • Guest

    The latter means you can dodge the IP block just by using the internet somewhere else or by power-cycling your router or using a proxy.

    These recurring objections are why I don’t like the “Mabus Problem” name. Mabus is a particularly obsessed person who would be willing to put in this kind of effort to continue his harassment. I doubt most of the people we’re talking about are so dedicated, or are even technically inclined, and the approaches you’re suggesting would be far more effective with them than with the Mabuses of the world.

  • Azkyroth

    We need a new law, like Dunning-Kruger, Godwin, and so forth.

    “Any statement calling attention to a problem and proposing general directions for solutions will attract objections to the effect that the problem is insoluble and society shouldn’t bother trying, which are usually refuted by information contained in the original post.”

  • David Simon

    That’s not as much of a difference as you think. Spammers also try to personalize their messages as much as possible, by adding randomized text and such, because a mass broadcast is easier for filters to flag.

    I’ve done some work with English-language spam detection, and the trick to defeating this kind of spam is that the “randomized text” is usually not very random. It’s either non-grammatical gunk that tends to follow a template or be otherwise distinguishable from English text, or else it’s snippets randomly selected from some large corpus such as the Gutenberg library. Also, the random additions can’t detract too much from the actual spam payload text, or the spammer gains no benefit; as long as their ad text or link is possible to pick out from the noise, the message is still detectable.

    Spammers have to make these compromises because of the high volumes they work in, and that leaves them vulnerable. It’s unfortunately a different ball game from detection of specifically targeted harassment, which can have a much higher ratio of human attention to message volume. :-(

    I tend to think that reputation-based systems are a more promising avenue than content detection. That’s actually where a lot of anti-spam work has been successful too; rather than trying to fight spammers on an escalating pattern-by-pattern basis, the impetus is instead put upon legitimate email sources to identify themselves as such in a hard-to-spoof way.

    Regardless, I think your ideas of word blacklists and IP bans have merit. It’s certainly possible for a persistent harasser to work around these mechanisms, but I think they would be enough to cut down the amount of nastiness by a fair degree in many cases.

  • Azkyroth

    Reputation-based systems can also be gamed, particularly with the prevalence of misogyny and analogous bigotries.

  • David Simon

    That’s true. One possibility is the use of reputation nets; that is, if Alice rates Bob highly, then Bob’s rating of Carol has a greater effect on whether Carol’s message to Alice is filtered, and the inverse applies if Alice rates Bob lowly.

  • BlueGuyRedState

    Most of your suggestions are worth trying, but IP blocking is easily circumvented and more likely to impact innocent bystanders than it is to even slightly inconvenience a motivated jerk. And it should be obvious that most of these jerks are highly motivated — motivated enough to spend a few seconds with Google if they don’t already know how easy it is to bypass IP restrictions.

    I think the real problem is the anonymity of the Internet coupled with the lack of any punishment or “cost”. In other words, as you already pointed out, the easy availability of “disposable” accounts.

    Ideally, I’d like to see ALL public discourse easily attributable to a “real person”. But I do realize that there are situations where some anonymity is helpful to allow people to speak freely.

    My own nym is a case in point. If my Kentucky neighbors (and even some of my “friends”) knew that I was an outspoken atheist, I would pay a social price.

    Nevertheless, I do believe that we would all benefit if there was a good mechanism that could discourage pointless vitriol, intimidation and threats of violence. And it should go without saying that legitimate criticism and disagreement should not be prevented.

    So let me propose something radical — REAL MONEY. If everyone had to put up some kind of deposit before they were allowed to make comments, I think that bad behaviour could be reduced dramatically.

    I don’t mean “pay per comment”. I’m talking about fining the originators of socially unacceptable comments as determined by some combination of readers and site administrators. In other words, if you get downvoted sufficiently, you lose your “deposit” and can no longer comment.

    Obviously, site administrators have the final say and should be able to arbitrarily “enforce the rules”. Conversely, a smart admin will not ban legitimate, respectful criticism because controversy drives page views.

    Certainly, this approach will need to be fine tuned, but it seems to me that we will never solve the problem without some accountability mechanism. We wouldn’t even need to sacrifice anonymity. You could make your deposits via BitCoin or something similar, just so long as there is an actual penalty for antisocial behaviour. Especially if the monetary penalty is forfeited directly to the victim of the abuse.

    Twitter, Facebook or anyone else could implement this immediately, and even make a profit if they chose to keep, say 50% of any “fines” levied. Any users willing to conclusively verify their “real ID” wouldn’t even need to put up a monetary deposit because they are already accountable by definition. Lurkers, i.e. people who just read the comments and never make them would never have to make any deposit at all.

  • Psycho Gecko

    There’s also some really screwed up stuff coming out about the harassment of Zoe Quinn. After she posted some screenshots from a 4chan IRC channel where they were coordinating to astroturf GamerGate, they accused her of cherrypicking and released a huge transcript from the channel. No question now that it was about harassing Zoe Quinn instead of outrage over journalistic ethics. Unfortunately for them, the transcript makes them look even worse.

    http://wehuntedthemammoth.com/2014/09/08/zoe-quinns-screenshots-of-4chans-dirty-tricks-were-just-the-appetizer-heres-the-first-course-of-the-dinner-directly-from-the-irc-log/

    It reminds me of that “Anatomy of a Men’s Rights Activist” story Vice did, only no way can people accuse anyone of strawmanning the MRAs on this one.

  • James Picone

    Further attacks on this kind of thing:
    – Deliberately attempt to contaminate the Bayesian filter by sending abusive messages that contain high-value words. “Rah rah swearwords rah misogyny”, a whole bunch, and attempt to train the filter into recognising ‘misogyny’ as a word likely to occur in a abusive messages. Inflating the false positive rate doesn’t benefit spammers, and they don’t get to pay much personal attention to messages or coordinate. Inflating the false positive rate here can make the filter worthless.

    – Don’t send abusive messages, send useless ones. Spammers are hamstrung by requiring a payload – they need to link or mention the thing they’re selling or there’s no point. Being a jerk online has no such restriction. For example, a bot that just reposted stuff from a year ago on /r/feminism, with no particular order, is going to be extremely disruptive and won’t trigger that kind of filtering at all. Detecting a whole bunch of duplicate messages will, but then you just grab a wider corpus – say, anything in Google Books, the contents of /r/politics, whatever.

    I really don’t think you appreciate the amount of effort that would have to go into trying to fix this and exactly how leaky the end result is going to be. Building a basic Bayesian filter and adding an option to turn it on is maybe a couple of weeks work for one person, including some testing. Even that basic filter affects areas as diverse as UI for viewing messages, configuration, saving and restoring user preferences, whatever notification mechanism is around, etc.. As it goes on and abusive users get used to the existence of the filter you need hack on hack to keep it useful. Block messages with just a link and uncertain abusiveness. Block messages with just a link. Make sure the way the UI works doesn’t reveal whether a particular message got through the filter or not to the person sending the message. Some kind of heuristic to prevent gaming the filter to make useful words appear abusive (very high weighting to messages marked as useful, very low weighting for reducing messages already in the very useful territory?). Detect and block several similar messages from one user by some metric, even if they would otherwise get past the filter. This thing is an ongoing problem just to keep it basically functioning. It really, really as simple as you think it is. Keep in mind that the solutions you’re suggesting only half-work for spam as it is – sure, you don’t see spam that often, but it’s still some ludicrously large proportion of the internet’s email traffic.

  • James Picone

    You may find https://craphound.com/spamsolutions.txt to be interesting reading. It’s a checklist commonly used at slashdot whenever anybody says something like “Well obviously to deal with spam we need to X”. Some of it is applicable here, although the concern about client filtering not reducing bandwidth usage obviously doesn’t apply.

    Classifying messages is a fundamentally hard problem. Remember, it /hasn’t worked for spam/. How much spam is there in your spam folder in gmail right now? Why is it still being sent if it doesn’t make money? Implication: Some people are still getting it, reading it, and clicking the links. And that’s with people taking a scattershot approach, who have a payload they must include, and are probably selling one of like four different stereotypically sold-via-spam products. Adam is advocating trying to take that heuristic and apply it to a situation where there’s a swarm of arseholes trying to flood someone with messages so there’s much more personal attention per message and it’s intended for one user, where there’s not necessarily a payload (being disruptive is nearly as good as being insulting), in fields where small messages are the norm.

    Just because it’s easy for you to do it doesn’t mean it’s easy for a computer to do it. Just because Google can do it in one context, kind of, doesn’t mean that Twitter can. Computers aren’t magic. businesses have limited resources, and natural languages aren’t context free.

    “Raped teen ignored by police [link]” is useful. “The police ignore me raping teens like you” is abusive. “[name] lies about sexual assault statistics” is useful. “You’ll be a sexual assault statistic” is abusive. “Going out tonight?” from a friend is useful. “Going out tonight?” from a troll is abusive. Good luck getting a computer to figure out the difference.

  • With respect, you’re looking at this completely the wrong way round. You’re operating under the assumption that any blocking technique with a more than trivial number of false classifications is useless. But the goal of harassment is to frighten people off social media entirely – and all too often, it succeeds. Any filtering method that “only” screens out, say, 50% or 75% of the bad stuff, even if it’s less than perfect, will be a huge savings in terms of time, effort, and emotional energy that would otherwise be spent on dealing with this crap. That could well make the difference between some people continuing to use social media at all, versus deciding that the toll it takes on their mental health is just too much.

    Here’s an example just from today: My friend Amanda Marcotte has turned off her Twitter mentions feed completely because it’s so inundated with abuse and misogyny. In our terminology, she’s using a method with a false-positive rate of 100%! Why? Because Twitter doesn’t offer any useful tools. A Bayesian technique that screened out the abuse but flagged, say, one in every ten good messages as bad, would still be a major improvement.

  • Deliberately attempt to contaminate the Bayesian filter by sending abusive messages that contain high-value words. “Rah rah swearwords rah misogyny”, a whole bunch, and attempt to train the filter into recognising ‘misogyny’ as a word likely to occur in a abusive messages.

    This is not a problem for a well-trained Bayesian filter, as I’m sure you know. It just has to be trained on what constitutes “good” as well as “bad” messages. Acceptable terms can appear in either type of message; bad terms only appear in bad messages, and constitute incriminating evidence even if neutral or good terms are also present.

    Again, I think, you’re fundamentally overestimating the enemy. You seem to think that harassers are a nefarious army, willing to invest unlimited effort into getting around filters by any means necessary. There are a few serious obsessives, granted, but the vast majority of trolls are lazy, ignorant and immature. They harass because it’s easy and costs nothing. Put barriers in their way, even modest ones, and I predict most of them will find something better to do. See my comments to crashfr0g that draw the analogy with suicide nets on the Golden Gate Bridge.

  • Silent blocking would probably be a useful feature in general – being able to block twitter users without making it clear to the blocked user that they’re blocked. Messages from Eve to Alice just don’t get shown to Alice. That might already be how it works, I don’t know, I don’t use Twitter much and haven’t ever needed the admin tools, I don’t use Reddit either.

    Yes, Reddit and Twitter both already support this. On Twitter it’s called muting, on Reddit it’s called shadowbanning.

  • silentsanta

    I don’t like the name ‘Mabus problem’ for the online harassment, vilification, threats and attempts at blackmail people suffer is an extremely wide issue, one that is particularly pronounced for people in historically disadvantaged groups including women, ethnic and sexual minorities, as well as religious minorities (including but not limited to skeptics and atheists). But sadly, the whole Atheism+ movement and the reaction to it has made it clear that there are a sizable amount of people who identify as atheists who believe it’s fair game to troll, insult, threaten and harass their opponents into silence.

    I don’t wish to dismiss or minimise what Mabus did or does, nor be cavalier about his threats but he is a single individual who is quite clearly mentally ill; by contrast the harassment faced by women and ethnic minorities is coming from a huge number of people who have no such excuse for their behaviour.

    Further, women, sexual and ethnic minorities are frequently the targets of these attacks, but it seems unlikely (from my own experience anyway) that they engage in these activities as much. Atheists and skeptics have sadly shown themselves to commonly be perpetrators as well as victims of this behavior.

    So what I mean to say is that if the phenomenon is to be named, it should be about
    1. casting it as a societal problem, acknowledging the widespread nature of the perpetrators, rather than identifying a mentally ill individual, and
    2. naming rights should be from a canonical example targeting women speaking out, or ethnic minorities speaking out, or gay people speaking out, (or all three), to make sure the name acknowledges those who pay the highest price in a society that treats this behaviour as acceptable.

    just my 2c as a white/cis/straight/atheist male who has never had to deal with this shit.

  • crashfr0g

    Which social media sites have tried these measures and found that they didn’t work?

    Um, pretty much every blog with comments? You seem to think that gross and trollish comments and messages is a new problem, but actually this is decades-old and the solutions you suggest have been well-explored. If preventing your users from harassing each other was as simple as training a Bayesian model, WordPress would install with one by default. Discus would advertise theirs.

    I’m predicting that social media harassment works the same way

    Don’t your own examples disprove that? Making a video game is hardly an “impulsive, spur of the moment decision”, yet that’s exactly what Anita Sarkeesian’s attackers did. Trolls are willing to put an enormous effort into trolling because the “costs” are actually the benefit – they’re enjoying all the work they’re putting into it.

    Kathy Sierra’s programming skill is irrelevant unless she worked at, and more importantly had policy-making authority at, the social media companies through which she was receiving the most severe harassment.

    What? How do you figure that? Social media API’s are invariably open and extensible; if it was easy to filter Twitter using Bayseian machine-learning, there was nothing stopping Kathy Sierra from using it to filter her Twitter messages.

    Indeed, if these solutions are so easy there’s nothing stopping you from doing it, too; you hardly need to be a policy-maker at Twitter to write a Twitter client that filters based on a Bayesian model.

  • crashfr0g

    But you don’t have to do this on the back-end. Why do you think you have to do it on the back-end? If Bayseian filtering is the winning strategy, here, there’s no particular reason Twitter has to do it – you just have to write a Twitter client – the work of an afternoon at most, it’s a simple REST API – that does it. Anyone can do it – and would have, if it worked.

  • crashfr0g

    This is not a problem for a well-trained Bayesian filter, as I’m sure you know.

    It’s certainly a problem with any and all Bayesian filters, well-trained or not. Why do you think it’s not? (Your description doesn’t really jive with how Bayesian filtering actually works, incidentally – I think this is an area in which you’d benefit from a little more research.)

    You seem to think that harassers are a nefarious army, willing to invest unlimited effort into getting around filters by any means necessary.

    Doesn’t it kind of seem like they are? Changing accounts on the daily to have a fresh identity from which to reach their targets? Making entire video games to threaten and intimidate figures they don’t like? They enjoy the effort. It’s a combination of technical challenge and misogynist power trip; it’s not hard to imagine the sort of intellect that appeals to. Why wouldn’t they put the work in, if they enjoy it, and the ultimate result is that they will have communicated that no woman is truly safe?

  • crashfr0g

    I’m astonished by the degree to which you’ve mischaracterized these harassers. Don’t have the technical inclination to unplug a router and plug it back in? To Tweet from a smartphone? Surely that’s ridiculous.

    These people harass and troll because they like doing it. Expending the effort is the fun of it. Why would they stop if you just make it a little harder?

  • If preventing your users from harassing each other was as simple as training a Bayesian model, WordPress would install with one by default.

    It does; it’s called Akismet. I used it for several years, before moving to Big Think and then to Patheos, and I can testify that it works quite well. (I even trained it to filter out Markuze comments, when he was pestering my blog for a while; it got pretty good at recognizing his unique brand of trollery.) Disqus has something similar.

  • crashfr0g

    I was going to suggest a law that people with no engineering background usually overestimate the effectiveness, and underestimate the effort, involved in the engineering efforts they propose (for others to do, natch), but that actually is the Dunning-Kruger effect.

  • crashfr0g

    You’re talking about the anti-spam features, though, not any targeted anti-harassment features. Women still get harassed on Discus blogs and blogs that run Akismet, pretty much with impunity except for human moderation. And it’s cool that after a period of weeks or months and a training set of potentially hundreds of messages you were able to train Akismet to filter out a single commenter, but that’s hardly a scalable solution, now is it?

    Again, I think you’ve allowed your lack of software development background to lead you to a naive conclusion about how easy this is going to be. If it’s so easy, why not spend an afternoon and write up a Twitter client that does it? SciPy has some great machine-learning libraries.

  • crashfr0g

    I think maybe you misunderstand how IP’s work. Nothing stops your ISP from assigning your home router an IP address they had previously assigned to the coffeeshop. “Private IP address” just usually means “a network-internal IP usually assigned by a NAT router.”

    Aside from a small number of infrastructure-critical servers, almost no host has a static IP anymore. Your phone is picking up a new IP pretty much every time you come out of the subway. We’re pretty much at the end of IP blocking as an effective moderation tool; have been for a while.

  • Azkyroth

    And who is it you think has “no engineering background?” As I recall Adam works either as a programmer or something similar, and I do mechanical, (IE actual, real, not, “it’s technical so let’s slap “ENGINEER” on it :D) engineering for a living.

    What was that you were smarming about Dunning-Kruger?

  • Al Petterson

    And even if it only screened out 90% of the abuse at any given time – so long as it continued to learn from new abuse flags – would do a lot to remove the overwhelming aspect of trollspew.

    But part of me thinks Brin has it right – that we need to admit that privacy isn’t going to be a right anymore; everything that you say and do online will be autotagged with your personal information.

    That’s horrible on its face, and will shut down a ton of important resources we have now (people anonymously reporting or discussing abuse, for example), but we’re reaching the reality where anyone with sufficient resources can find out who you are anyway (unless you in turn devote large resources to protecting your privacy), and so the question will be whether you want only powerful people and organizations to have privacy and be able to violate yours, or whether everyone can see everyone and the power’s symmetric.

    Because if trolls weren’t anonymous, there would be other means of dealing with them. And if a few actually got dealt with that way, there might be fewer trolls.

  • Al Petterson

    I propose it be Azkyroth’s Law, having not seen it expressed quite that way before.

    (Powerfully reminiscent of the argument that we can’t solve global warming because everyone in China wants a Buick, so don’t bother doing anything.)

    I find that “FUDdite” is already in the Urban Dictionary, but it’s certainly a descriptive term for the people who pronounce everything unsolvable so why bother trying.

  • Al Petterson

    I also visit a lot of blogs that have a lot of traffic and few trolls.
    Unfortunately they all have two things in common: (1) they have a moderator who puts regular effort into at the banhammer, and (2) they’re all fronted by white males.
    So I do have a suspicion that both you and I might be underestimating the difficulty simply because the success stories we see (I’m thinking of slacktivist, balloon-juice, makinglight) are susceptible to being overwhelmed by the sort of person who doesn’t bother to overwhelm white men.

  • Al Petterson

    (I note, in passing, that “socialism” cannot be uttered in the comments of many blogs, because it contains the name of an ED drug.)

  • Guest

    I would disagree. On the scale of harassment from…

    *Actual murderers/attempted murderers
    *People who cause physical injury
    *people who physically confront others
    *people who avoid confrontation, but harass in person (e.g. Shout at pedestrians from a moving car)
    *stalkers
    *snail mail/notes/graffiti harassers
    *email/Internet harassers

    Internet harassers are on the low end of the effort scale, and aside from the famed Mabus, haven’t demonstrated they’re willing to incur a temporal or social cost to continue their actions.

  • Verbose Stoic

    Someone doing a Let’s Play of an MMO had the game refuse to accept his profile because it contained the phrase “hero in” … which without the space is “heroin”.

    Computer pattern matching algorithms can be frighteningly stupid.

  • Again, I think you’ve allowed your lack of software development background to lead you to a naive conclusion about how easy this is going to be.

    For the record, I have a master’s degree in software engineering, and my day job is writing code for a New York technology firm. I think it’s safe to assume that I understand what would be needed for a solution to this problem.

  • Martin Penwald

    I disagree with you on the IP address thing. For example, 3G/4G network providers give their customer’s device a local IP address (typically 10.x.x.x), then a lot of different customers get access to Internet through NAT, with the same exposed IP address. IPv6 would have been a better solution, but more expensive, so we are stuck with this stupid IPv4 NAT.

    Moreover, I am not convinced that big social networks have an interest in moderating their users.

  • Ryan

    These sound like good options. As a gamer, I’m quite skeptical of some of the claims Anita Sarkeesian has made in her videos, especially with things being taken out of context and lack of evidence for wide-ranging statement.
    This makes it harder to critique rationally or cordially without being given the tar-brush of the trolls that have attacked her

  • Azkyroth

    Internet harassers are not homogeneous. While there doubtless is a core group of fanatics who would consider harassment no matter what, a substantial proportion of them are jumping on the bandwagons of their toxic little communities and/or looking for cheap, easy kicks, and have limited motivation or persistence. Removing them from the equation will make the problem more manageable both by reducing the absolute volume of harassment and undermining the argument that “aww, they’re not really serious!” that authority figures use to deal with it, since there will be a selection bias towards those who are really serious with regards to inclusion in the smaller scale harassment that continues once filtering mechanisms are in place.

  • Azkyroth

    So let me propose something radical — REAL MONEY. If everyone had to put up some kind of deposit before they were allowed to make comments, I think that bad behaviour could be reduced dramatically.

    I don’t mean “pay per comment”. I’m talking about fining the originators of socially unacceptable comments as determined by some combination of readers and site administrators. In other words, if you get downvoted sufficiently, you lose your “deposit” and can no longer comment.

    So basically only people with a fair amount of disposable income will be able to participate in the common discourse even online.

    This won’t end well.

  • BlueGuyRedState

    “So basically only people with a fair amount of disposable income will be able to participate in the common discourse even online.”

    How do you figure that? Did you even read the entire suggestion? Especially the last paragraph? The only people ever affected would be those who want or need an “anonymous” account.

    Even then, only the abusive jerks would be penalized. Simple disagreement or criticism should be tolerated and even encouraged by any smart site admin. As I said, legitimate controversy drives page views. Only when someone makes threats or is extremely abusive should they lose their “good behavior guarantee”.

    A $10 or $20 or even $50 deposit shouldn’t be a huge entry barrier to anyone who can afford the equipment and bandwidth to access the Internet in the first place. The deposit could even be refundable after a certain length of time or accumulation of “karma”. The goal is to enable civilized discourse while minimizing bad behavior.

    Of course I realize that nothing will prevent all abuse, but this suggestion is simply one tool that can be tried. I think it is valuable because a disproportionate number of problems are caused by the ready availability of “anonymous accounts” and, at least at the moment, there is literally no penalty for abusing these throw-away accounts. Site admins can do nothing except play a frustrating game of “whack-a-mole”.

    I have no objection at all to simultaneously trying most of the the other suggestions as well — but please realize that no one thing will provide a complete solution. Every technological solution (including my own suggestions) will inevitably produce an “arms race” as some people will try to get around any restrictions.

  • CandideThirtythree

    Lots of MMOs have addressed the problem of trolls and you rarely see the acres of potty mouth posts that were SOP for game forums in the beginning.

    They have managed to block gold spammers, who have hundreds of thousands of accounts with names like “ogvxsrkhsw” names and accounts that are randomly computer generated in sweatshops in china then played on an endless loop in the chatbox of even unpopular games.

    For nongamers, gold spammers are people who sell in-game currency to players, usually all they do is steal the credit card information of gullible children who have swiped mommy’s credit card to buy fake money so that they can buy digital merchandise in a virtual world. They sell it by spamming the in-game chat in online video games with programs that repeat the same message thousands of times a minute.

    There have been documented cases of people in China being literally chained to a bank of computers and forced to play video games to collect this digital currency for unscrupulous people. So we are talking the most desperate of people, who’s every waking moment is devoted to trolling/spamming and yet most game companies have managed to get a handle on it so why can’t Twitter?

  • CandideThirtythree

    NO, it is like sites making you use a cell phone number to retrieve your account activation code, you cannot even open an account without that cell phone because there is no way to get the activation code except with the cell phone attached to that number you gave them to start the account.

    So then your cell phone number is what is used to block your account, then you would have to have access to a different cell phone for every account you start.

    That would get expensive.

    The problem is that the site would have to have moderators and people to handle customer complaints about getting banned.

  • bruce bartup

    I believe there is more to worry about annd more to hope for here.

    As wwell as a threat in themselves hate commentary is also a rich psycho-social data in a convenient form for analysis. Often with interpretive meta data attachhed – ‘lkes’. Potentiallly?

  • Azkyroth

    A $10 or $20 or even $50 deposit shouldn’t be a huge entry barrier to anyone who can afford the equipment and bandwidth to access the Internet in the first place.

    You need to talk to more people. Seriously.

  • Azkyroth

    What.

    This looks like spam but there doesn’t seem to be a link attached…

  • BlueGuyRedState

    Your “arguments” MIGHT be more convincing if you could provide evidence that similar suggestions have already been tried unsuccessfully, or at least be willing to explain the logic that you used to reach your conclusions.

    Internet naysayers are a dime a dozen. At least be willing to propose some constructive solutions. Or else be specific about your objections if you wish to actually have a productive conversation.

  • BlueGuyRedState

    Not sure I understand your objection. So far, EVERY proposed solution entails a fair amount of overhead.

    And it would only get expensive if someone regularly get banned because they insist upon making rape or death threats or are otherwise extremely antisocial. As I have said repeatedly, the Internet thrives on legitimate controversy, so “normal” disagreements would never be impacted.