Why I may not donate to MIRI/CFAR in the future

I spent much of the last two days writing a follow-up to my previous post on the LessWrong community’s crackpot problem. But then I looked over what I’d written and thought better of a lot of it.

I stand by what I wrote in my original post, and as a member of the effective altruism movement, I think it may sometimes be worth risking offense in order to give an honest assessment of non-profits and the cultures that surround them.

However, much of the evidence I could give for my assessment is from things people have told me offline, in e-mails, or on Facebook, contexts where I think people could reasonably expect not to get what they say signal-boosted on a very public blog. I won’t ignore the information I have in forming my own opinions, but I recognize that broadcasting all of that information may not be wise.

(No great scandals here. Just a variety of moderately embarrassing stuff that arguably isn’t all that surprising if you’ve followed the LessWrong community online. It was just different for me seeing it up close, I guess.)

I can say that I think the problems with LessWrong’s culture run deep. There are things in Eliezer Yudkowsky’s “Sequences” that didn’t hit me as problematic when I first read them, but now do, big time. His “Correct Contrarian Cluster” post, for example.

I tried to express a lot of what I think is wrong with LessWrong’s culture in a post on LessWrong a few months ago, titled “Self-Congratulatory Rationalism”. One thing in that post may not have been clear—the point was not to argue for humility or self-assurance per se, but to advocate being less dismissive of those not in your in-group, while also not taking for granted the sanity of those who are in your in-group.

Though he’s subtle about it, I think Robin Hanson has done a good job of pointing out the problems in the LessWrong meme cluster. Here’s a recent example. On Twitter, I suggested that his main point—be willing to just trust others’ judgments—was sound, but his “Don’t be rationalist” framing was off. Shouldn’t knowing when to trust others be part of rationality?

In response, he pointed out that for many people, “being ‘rationalist’ means not needing to listen to those not ‘one of us.’” Unfortunately, I think Robin’s right about that.

Alexander Kruel also makes good points. I wish I’d grokked his worries about LessWrong when I first encountered him online. He gets many Bayes points, as Eliezer would say.

I still like many of the people I’ve met through the Bay Area LessWrong community on a personal level. On the whole, they’re every bit as smart as advertised, and it’s a lot of fun to get to hang out with such smart people on a regular basis.

Unfortunately, I think the correlation between intelligence and rationality is weak at best. Also, that having a fun, quirky subculture, and a movement that actually gets stuff done, are goals that are somewhat in tension with each other.

How does this impact how likely I am to donate to MIRI, and its sister organization CFAR, in the future?

Let me start by summarizing my thinking about MIRI as of roughly late last year: MIRI’s current focus is on doing research on AI safety directly. I agree that this is potentially important work.

I’ve always been skeptical of MIRI’s idea that they’re going to try to be the ones to create a superintelligent AI directly. But Luke and other people at MIRI seem to be trying to make sure their work will be valuable to other researchers if someone else ends up creating the first super-AI.

This is a very good thing. In fact, I first donated to MIRI because Luke convinced me that the value of their work did not depend on any very specific assumptions about AI being right.

In general, I think Luke has done great work as executive director of MIRI, and MIRI is in much better shape than it would be if not for his work. He also seems to at least be pulling MIRI and LessWrong in the right general direction with respect to the community’s distrust of mainstream experts.

But I’m no longer sure it’s enough. I don’t have much direct evidence on what effective strategies for AI safety research might be. Donating to MIRI means trusting them to make those judgments. Though making a judgment like that is extremely complicated, seeing big problems with the overall LessWrong memecluster tip me towards not being able to trust them with those judgments.

I also worry about MIRI becoming isolated from the academic mainstream. The beauty of science is that it uncovers the truth without depending on the rationality of any one scientist. For examples, see Alexander Kruel’s list of highly intelligent and successful people with weird beliefs—almost all of them highly respected scientists, mathematicians, and philosophers.

If Eliezer Yudkowsky were an ordinary academic, his mix of very good and very bad ideas wouldn’t be much of a problem. Let the rest of the scientific community sort it out.

As it stands, can we count on a similar process happening with MIRI’s research? I understand MIRI is making an effort to interact with the academic mainstream, bringing in mathematicians from the outside, sending papers to conferences, but again, will it be enough? I don’t know.

Currently, I have no plans to further donate to MIRI in the future. I have a monthly donation set up for the Schistosomiasis Control Initiative. I’ve also been tentatively planning on donating to the Centre For Effective Altruism (CEA) around the end of this year (no good way for an American to set up a recurring donation AFAICT).

But this may change in the future. One worry about CEA: I haven’t seen the community around CEA in Oxford up-close, the way I’ve seen the Bay Area LessWrong community up close. Would a closer view of CEA leave me with similar worries about them? Whatever happens, it seems likely I’ll continue to change my mind about which charities to donate to in the future.

  • Robby Bensinger

    Your reasoning here makes some sense. In particular, you might want to look into funding academic organizations doing things similar to MIRI, if you’re worried that their research (and therefore their perspective) is being under-funded and under-noticed compared to MIRI’s. That could serve the dual purpose of funding AI requirements research or forecasting, and of helping give you data with which to assess MIRI.

    Your main worry is that MIRI is isolated from academia, and that you don’t have outside sources to consult for AI strategy. As far as I can tell, the academic center for AGI forecasting and strategy is currently the Future of Humanity Institute. Bostrom and Yudkowsky agree on a lot, but in a number of cases that’s because Yudkowsky came around to Bostrom’s position, not necessarily the other way around. FHI and MIRI also seem to be very well-connected, so I can’t say I agree that MIRI is isolated. Possibly CSER is a better organization to support than FHI, though, if you’re already familiar with the points made in e.g. Bostrom’s new book Superintelligence, and still want more outside perspectives on the strategic questions.

    My own experience with Alex Kruel is that he throws out large numbers of arguments in rapid succession, and when any one of them gets scrutinized he changes the topic or is otherwise less-than-responsive to the new evidence. I end up sometimes using him sort of as a random argument generator: I don’t trust his big-picture assessments or his calibration regarding which arguments are good and which are bad, but if I’m discerning I can find some new thoughts I haven’t seen elsewhere and end up at productive realizations as a result.

    • http://patheos.com/blogs/hallq/ Chris Hallquist

      I was confused by a lot of what Alexander Kruel says at first. In retrospect, it would have been helpful if, rather than looking for individually knock-down arguments, I’d seen each argument as “potentially worrying sign; reason to update a bit towards being worried about MIRI.”

      • Robby Bensinger

        Yeah, updating incrementally is indispensable, and it’s not easy. I don’t think I’ve personally run into the problem of dismissing everything Alex has to say out-of-hand for being inconclusive. It almost always seems obvious to me that Alex’s arguments vary in quality a lot. His worst arguments aren’t an excuse to dismiss his best ones; equally, his best arguments don’t demonstrate that his average argument is worth taking very seriously.

        E.g., his arguments on http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care were sometimes interesting, and occasionally correct. But the overall impression was not “Kruel is chipping away at the opposition bit by bit”, but rather “Kruel is throwing everything against the wall to see what sticks, but has a poor grasp on the structure of MIRI’s arguments (e.g., on very basic points like ‘the seed is not the superintelligence’ and ‘having the preference to be a good spellchecker is different from having the ability to be a good spellchecker’)”. He frequently skips out on doing 5 minutes of basic research or self-criticism, even when that should suffice to do away with a claim of his. So it can be exhausting to wade through all his mistakes to find the interesting points. Yet I don’t see him heavily qualifying his arguments to signal uncertainty or disclaim, ‘I’m just throwing this out there to see if there are problems with it, I have no idea whether it’s right or not’.

        This was the conclusion I reached when both MIRI and Alex’s arguments were very new to me, and I’ve seen it borne out pretty regularly by subsequent interactions with each, though not uniformly so.

        • http://kruel.co/ Alexander Kruel

          > … but has a poor grasp on the structure of MIRI’s arguments…

          I find MIRI’s arguments to be extremely convoluted and based on vague premises that I reject taking seriously. The result is often that people associated with MIRI don’t understand my arguments, because they are made in a different context, or based on different assumptions.

          To use an analogy: I don’t argue from the assumption that an autonomous car will be able to traverse a city without crashing, and then pose the question of how we can make sure that it will reach the correct destination. I do not discriminate between programming a car to autonomously drive, without crashing, and programming it to not drive its passengers into the ocean, if they asked it to drive them to the nearest seafood shop.

          Let’s analyze the very first sentence of your post you link to above:

          If an artificial intelligence is smart enough to be dangerous, we’d intuitively expect it to be smart enough to know how to make itself safe.

          This is already really weird, on many levels. To be able to create an AI smart enough to be dangerous requires
          us to learn a lot about how to make software behave as expected, without crashing. In other words, it requires us to become much better at encoding our intentions.

          An autonomous car that maximizes the number of people it can kill, by running them over, is a very unlikely failure mode. Because the hard part is to get the autonomous car to autonomously drive, and detect people, without destroying itself. The hard part is NOT to prevent it from maximizing the number of people it runs over.

          All the arguments that MIRI makes, e.g. that reality
          does not bite back when it comes to values, are completely irrelevant in this context.

          Seed AI? Intelligence explosion? Ridiculous. I
          reject these ideas. Give me hard evidence or an overwhelming consensus among experts, and I will take these ideas seriously. Otherwise they are just more weird ideas that might superficially make sense but which are
          almost certainly wrong.

          A handful of people like Stuart Russell and Jürgen Schmidhuber making grandiose predictions about AI are just noise, as long as they are not even able to create AIs that can navigate autonomously in a real-world environment and survive real-world threats and attacks with approximately the skill of an insect (and no, Schmidhuber’s robots fall far short of this) .

          What will happen is that AI will be incrementally improved, while any product that behaves unexpectedly will be abandoned.

          • Robby Bensinger

            > To be able to create an AI smart enough to be dangerous [...] requires us to become much better at encoding our intentions.”

            It requires us to become much better at writing programs that perform some behaviors we want them to (e.g., the behavior ‘test plausible hypotheses’). But you consistently conflate the skill ‘write a program that can do some things we want it to’ with the skill ‘write a program that does only what we want it to.’ This is part of why I say that you don’t understand MIRI’s arguments, and for the most part aren’t intersecting with them.

            Some of MIRI’s arguments may be ‘extremely convoluted,’ but the ones I’ve seen you responding to are mostly pretty simple. This again makes me think that you aren’t understanding them. (If you’re confused by something, it’s easy to fall into the trap of assuming that the thing is more chaotic or complex than it really is, rather than recognize that the jumble is in your model of the thing. This is really common; it’s not a personal failing on your part.)

            > An autonomous car that maximizes the number of people it can kill, by running them over, is a very unlikely failure mode.

            It’s similarly unlikely that a hammer or a stapler will spontaneously turn into a domain-general optimizer with a lust for blood. A car lacking in general intelligence or complex causal inference isn’t relevantly similar to an AGI.

            > Seed AI? Intelligence explosion? Ridiculous. I reject these ideas.

            I haven’t seen any knock-down argument against either idea. There is no consensus for or against them, to my knowledge. Expressing extreme confidence in the impossibility of intelligence explosion doesn’t seem to put you in a much better position than MIRI, when it comes to conforming to expert consensus; there is no consensus in any direction about the microeconomics of cognitive reinvestment, though there are scholars (e.g., Robin Hanson) who lean more toward your view than toward MIRI’s.

            > A handful of people like Stuart Russell and Jürgen Schmidhuber making grandiose predictions about AI are just noise, as long as they are not even able to create AIs that can navigate autonomously in a real-world environment and survive real-world threats and attacks

            I find this response a bit hard to take seriously. Dismissing the views of highly respected academics as “just noise” again makes me think that you’re suffering from overconfidence biases. You might want to try some exercises, like arguing both for and against your position, or paraphrasing others’ views and asking them if you’ve accurately represented what they said, as a sanity check on your ordinary approach in dialogues.

          • http://kruel.co/ Alexander Kruel

            You might want to try some exercises, like arguing both for and against your position, or araphrasing others’ views and asking them if you’ve accurately represented what they said…

            I have done so often. Here is e.g. my primer on AI risks. So far I have never seen you paraphrasing my views correctly.

            But you consistently conflate the skill ‘write a program that can do some things we want it to’ with the skill ‘write a program that does only what we want it to.’

            No, I don’t. Show me one example of where an AI failed by doing something ingenious that it was not programmed to do?

            A car lacking in general intelligence or complex causal inference isn’t relevantly similar to an AGI.

            If intelligence is something that is independent of preferences, then what exactly is the problem with the analogy? If my calculator does currently work as intended, why would it stop doing so if you added intelligence to it? Why would an autonomous car suddenly diverge from its previous behavior? I don’t think it would. Because if all you do is to add intelligence, then it would have no preference to do something other than what it was programmed to do. It simply isn’t rational to take over the world in order to ensure that you are not stopped from doing whatever you are doing. If you are not programmed to care, then you won’t. And my calculator certainly does not care if it is destroyed. Intelligence won’t magically make it care.

            I haven’t seen any knock-down argument against either idea. There is no consensus for or against them, to my knowledge. Expressing extreme confidence in the impossibility of intelligence explosion doesn’t seem to put you in a much better position than MIRI…

            Haha!!! Nice try shifting the burden of proof here. I don’t need any knock-down arguments to dismiss Roger Penrose’s belief that consciousness is uncomputable, or Max Tegmark’s mathematical universe hypothesis. People who hold extraordinary beliefs need to provide strong evidence and make falsifiable predictions.

            Dismissing the views of highly respected academics as “just noise” again makes me think that you’re suffering from overconfidence biases.

            Not if they are outliers.

  • http://CommonSenseAtheism.com lukeprog

    > “becoming isolated from the academic mainstream”

    Just a comment on this one line. I’m pretty sure things are moving in the opposite direction. The academic mainstream becoming more interested in the challenges of AGI safety due to Bostrom’s book and Stuart Russell’s outreach. Also, MIRI is now starting to publish papers on its technical research agenda in the major conferences and journals, e.g. see our recent AAAI paper and my forthcoming article (w/ Bill Hibbard, due in Sept 2014) in the most-read computer science magazine, Communications of the ACM.

    • http://patheos.com/blogs/hallq/ Chris Hallquist

      Congratulations on those papers! Yes I understand MIRI is making progress on this. The question is will it be enough, what will happen in the long run. (Maybe “ending up isolated” would have been a better word choice.)

  • MNb

    “Self-Congratulatory Rationalism”
    It’s irrational to assume that you are (I am) rational. According to psychology no human being is rational, not in the meaning as used in your Less Wrong article.

    http://psychcentral.com/blog/archives/2011/01/31/are-we-rational-animals/

    • JohnH2

      That is sort of the point of what Less Wrong is supposedly after, getting rid of biases and becoming rational enough, supposedly to be able to frame the issue of “Friendly” AGI.

      Internal surveys at Less Wrong, and external research however shows that intelligence is not correlated with being more rational and attempting to deprogram biases actually leads to worse results than beforehand. Less Wrongians often say that isn’t true, but internal membership surveys show them to be objectively wrong.

      Or in their terms, go read The Sequences (their sacred text, hundreds of pages by their founder).

      • Romeo Stevens

        >attempting to deprogram biases actually leads to worse results than beforehand.

        Interesting, any pointers to some material on this?

        • JohnH2

          The funny thing is that the point is part of the Sequences at Less Wrong and further research is linked in the comments.

          There is repeated evidence though that “Bayesian” is not magically making them free from those same problems:
          http://lesswrong.com/lw/fp5/2012_survey_results/
          http://lesswrong.com/lw/jj0/2013_survey_results/
          but rather much worse than the baseline of the general population, as the literature on the subject actually suggests would be the case.

          • David Simon

            What is in those survey results that shows a lack of rationality?

          • David Simon

            Oh, sorry, missed the overconfidence bias test results in the middle. I take it that’s what you’re talking about?

          • JohnH2

            Yes.

      • MNb

        Yes, that’ was what attracted me. The thing is though, if you want to get rid of biases and become more rational, you need to recognize those biases (where you’re wrong) first. Over and over again. I found very, very little of this kind of self-criticism on that site. So I find it utterly uninteresting.
        I expected a website according to Richard Feynman’s First Principle, but found rather the opposite.

  • Tle

    I’m reading the sequences (related to rationality, biases, so on) on LessWrong. Is there a list of problematic posts or critiques of them? I can’t say I would realize if there’s anything wrong by myself.

    • http://patheos.com/blogs/hallq/ Chris Hallquist

      Biggest problems, I think, are the stuff that’s anti- trusting mainstream experts, and holding out promises that good rationalists should be able to do vastly better than traditional scientific rationality.

      Plus some general problems of overconfidence, and that even when Eliezer is making good points, you’re not finding out the arguments on the other side of the issue, so he shouldn’t be your only source for anything.

      • Tle

        Thank you. Do you know other sources that have the arguments on the other side?

        • http://patheos.com/blogs/hallq/ Chris Hallquist

          Unfortunately, a lot of the debates the sequences cover lack good popular-level presentations. Susan Blackmore has some okay popularish books on consciousness that cover the range of perspectives in the field. (One, Conversations on Consciousness, features interviews with a wide range of experts.) David Chalmers’ book The Conscious Mind is worth at least trying to read, though I don’t know how accessible you’ll find it.

          Topics like metaethics, or Bayesianism, or interpretations of quantum mechanics… I wish I knew what to tell you.

        • http://newstechnica.com David Gerard

          The comments often point out some obvious flaws.

          The Sequences are 99% science and philosophy popularisation and 1% actual weirdness.

          The 99% is pretty good. The main problem is the use of local jargon for things that already have names – this means that you’ll see an exciting new concept and you can’t just do a search and get 2000 years of discussion of the topic.

          The trouble with the 99% is that you go along with it and swallow the 1% along the way, and the 1% can get seriously weird.


CLOSE | X

HIDE | X