Why you should consider donating to the Singularity Institute (I did!)

I’ve been on the fence for awhile about whether the Singularity Institute is worth donating to. I alluded to my uncertainty in my post on Kony 2012. I finally made my first donation just now. Two things pushed me over the edge.

First is Luke Muehlhauser’s reply to Holden Karnofsky of GiveWell regarding donating to the Singularity Institute. A couple points that stood out were the argument that the case for worrying about AI risk does not depend on specific views of Singularity Institute folks, as well as Luke’s citation of an endorsement of the Singularity Institute by Nick Bostrom (who, in my opinion, does good work and probably knows what he’s talking about in these matters).

Second was just the fact that some Singularity Institute donors have pledged to match donations to the Singularity Institute for up to $150,000 for the month of July, which doubles the effectiveness of donations made between now and either when that goal is reached or the end of July, whichever comes first.

But I guess the real issue is not what put me over the edge, but what had me seriously considering donating to the Singularity Institute in the first place. The main reason that I think big changes are coming soon enough that they’re worth worrying about now.

Furthermore, I think even if the Singularity Institute doesn’t directly accomplish everything Eliezer Yudkowsky hopes it will, it’s work is (right now) having good effects by getting important ideas out there by means of things like the Singularity Summit and the academic papers Luke has been working on. I think it’s valuable to increase the number of smart people working on these issues.

This is not to say you definitely should donate to the Singularity Institute; I’m actually somewhat uncertain as to whether it’s a better place for your charitable donations than the charities recommended by GiveWell. But I’d advise considering it.

Link to the Singularity Institute’s donate page and page on the matching of donations. Link to GiveWell for comparison.

  • zzzzzing

    Is this for real or some kind of setup? How did someone who can say this…(from a previous PS of yours)

    P.S.–I’m trying to avoid moral exhortation here, because morality doesn’t motivate much. Instead, my point is that if your goal is to help people, there are better ways to do it than donating to Kony 2012.

    ..consider donating to this steaming pile of mental masterbation as if was actually helping people? If you have coin to spare and think that this kind of donation is on par with fixing a real problem in the current world, I am amazed and confounded.

  • http://kruel.co Alexander Kruel

    Since it isn’t even remotely clear that we will ever have to face unfriendly AI, the whole argument sounds about as convincing as the following:

    Dragon slayer: At some point a dragon will appear in your garage and you should better take that possibility seriously because you might be eaten.

    Dragon skeptic: Dragons are mythical creatures, I doubt that I will stumble upon a dragon any time soon.

    Dragon slayer: Given what we know about physics, dragons are possible. And since being eaten is extremely negative, even given the lack of any empirical evidence in support of the possibility of a dragon appearing in your garage, you should take that possibility seriously. Just think about your possible children and your children’s children and their children. The loss in expected value resulting from your death will be enormous. You have to save those people!

    Dragon skeptic: To refine my estimations regarding your dragon, what do you anticipate to happen before the dragon will appear in my garage, is there any possibility to update on evidence before the dragon appears?

    Dragon slayer: No, I don’t know enough about dragons to be more specific about my prediction. I will know when I see the dragon though.

    Dragon skeptic: Hmm. Could you then tell me what led you to believe that a dragon might appear in my garage and why it would be dangerous?

    Dragon slayer: We know that once upon a time huge giant reptiles roamed the earth. We also know that flamethrowers are technical feasible. Further, most giant animals do not care about human well-being, which makes them extremely dangerous. Well okay, elephants and whales are not dangerous but you can’t reasonably expect that most giant flame throwing reptiles are like that…

    Dragon skeptic: Ok, let’s assume such a thing is possible. Why would it appear in my garage? I mean, sure, evolution might result in such a thing as a dragon at some point but…

    Dragon slayer: I didn’t say it will happen tomorrow, I don’t like to talk about time frames.

    Dragon skeptic: I see. Then what do you suggest that I do?

    Dragon slayer: Man forges his own destiny. Give me money so that I can think more about dragons and forge a sword with which I can slay any dragon. I call it Excalibur.

    Dragon skeptic: Interesting. I’d like to see your credentials that you are a sophisticated blacksmith.

    Dragon slayer: I am afraid I can’t do that. A proof of my skills would involve the disclosure of dangerous knowledge! The kind of sword that I forge could be abused to slay humans more easily than dragons.

    • http://www.facebook.com/chris.hallquist Chris Hallquist

      After glancing at the blog post you link to:

      Have you read this paper by Bostrom, discussing the orthogonality thesis and the instrumental goals powerful AIs with many different end goals are likely to converge on? Do you at least know what I’m talking about in the previous sentence?

      Brief response to the blog post: yeah, you could program an AI to respect what’s implicit in our commands, theoretically. But doing that would require some work, not be an automatic effect of “intelligence.”

      (Well, actually, something with human-level mental abilities in every area would by definition have human-level abilities to read intentions. Though I’d personally worry some about an AI that has superhuman abilities in, say, technology design, war planning, and defeating computer security but sub-human abilities in reading human intentions.)

      • http://kruel.co Alexander Kruel

        I find the arguments in it unconvincing and to a large extent wrong.

        I don’t think that I have left anything unclear about how I think it is wrong in my post you link to.

        What most of the arguments along those lines of Nick Bostrom’s paper, or e.g. Omohundro’s ‘The Basic AI Drives’ to name another example, accomplish is that they make specific assumptions to yield the desired conclusions in a tautological way.

        If your premise is 1.) an expected utility-maximizer, that is 2.) pulled at random from mind designs space, 3.) capable of undergoing explosive recursive self-improvement, that 4.) follows hard-coded goals without the desire to refine them, that 5.) tries to take every goal to its logical extreme, whether that is part of the specifications or not, then you already answered your own question and arguing about drives becomes completely useless.

        I’m not sure there is enough support for even paying attention to this hypothesis.

        • http://www.facebook.com/chris.hallquist Chris Hallquist

          Okay, more basic question, what do you mean by “intelligence” in your blog post?

          • http://kruel.co Alexander Kruel

            By “intelligence” I generally mean whatever it takes to overpower humans by means of deceit and strategy rather than brute force.

            More specifically, by “intelligence” I refer to the hypothetical capability that is necessary for a systematic and goal-oriented improvement of optimization power over a wide range of problems, including the ability to transfer understanding to new areas by means of abstraction and adaption of previous methods.

            In this context, “general intelligence” means to be able to ‘zoom out’ to detect global patterns. General intelligence is the ability to jump conceptual gaps by treating them as “black boxes”.

            General intelligence is a conceptual bird’s-eye view that allows an agent to draw inferences from high-level abstractions without having to systematically trace out each step. It is a wormhole. General intelligence allows an agent to get from here to there given limited computational resources.

            (If we assume some sort of scenario where an AI takes over the world by means of advanced nanotechnology then we are merely talking about grey goo by other names.)

            But note that the above is not what I believe to be the most important ingredient of intelligence. Which is in my opinion the existence of complex values and motivation.

            The idea that a rigid consequentialist with simple values can think up insights and conceptual revolutions simply because it is instrumentally useful to do so is implausible.

            Complex values are the cornerstone of diversity, which in turn enables creativity and drives the exploration of various conflicting routes. A singleton with a stable utility-function lacks the feedback provided by a society of minds and its cultural evolution.

            You need to have various different agents with different utility-functions around to get the necessary diversity that can give rise to enough selection pressure. A “singleton” won’t be able to predict the actions of new and improved versions of itself by just running sandboxed simulations. Not just because of logical uncertainty but also because it is computationally intractable to predict the real-world payoff of changes to its decision procedures.

            You need complex values to give rise to the necessary drives to function in a complex world. You can’t just tell an AI to protect itself. What would that even mean? What changes are illegitimate? What constitutes “self”? That are all unsolved problems that are just assumed to be solvable when talking about risks from AI.

            An AI with simple values will simply lack the creativity, due to a lack of drives, to pursue the huge spectrum of research that a society of humans does pursue. Which will allow an AI to solve some well-defined narrow problems, but it will be unable to make use of the broad range of synergetic effects of cultural evolution. Cultural evolution is a result of the interaction of a wide range of utility-functions.

          • DSimon

            Alexander, you make a lot of assertions here about what is required for general intelligence, and I notice that these requirements (multiple entities, complicated value systems, memetic selection pressure) line up closely with what human society does.

            I agree that it makes the most sense to look at our one existing example of intelligence to determine what goes into it. But, do you have any specific reasons to believe that all these aspects are required, rather than just one approach out of many?

            Also, even if we take it as a given that an AI with simple values won’t be able to do all the different complicated things humans do, what’s to keep such an AI from figuring out something narrow but dangerous?

          • http://kruel.co Alexander Kruel


            If it doesn’t take complex values then how does it take a general intelligence?

            What is the important difference between an army of robot scientists (narrow AI) stumbling upon dangerous knowledge, which enables unfriendly humans to wreck havoc, and an artificial general intelligence?

            If there exist simple algorithms that can vastly increase the rate of discovery of unknown unknowns then those algorithms will be used by humans in the form of expert systems (tools) before they are combined into a coherent framework of agency that is then using those same algorithms to eliminate humans either as a side effect or because they pose a risk.

            If the proponents of AI risks claim that it doesn’t take complex values and motivation then how does it take general intelligence at all?

            I perceive it to be a very implausible outcome in which stand-alone algorithms are insufficient to pose an existential risk, yet the addition of some sort of artificial agency with narrow values, which incorporates those algorithms, is sufficient for them to sprout an abundance of novelty.

            The region between narrow AI scientists and AGI’s with complex motivations is very narrow. How is that region the most likely outcome? A region where tools are not quite enough but algorithms with similar complexity as the human brain are not necessary?

          • DSimon

            @Alexander, the reason to expect more risk (and more risk sooner) out of the singleton AI than a group of incautious AI-researching humans is that only the former has the capability for rapid recursive self-improvement.

    • http://kruel.co Alexander Kruel


      I don’t know of any good reason to believe that rapid AND vast recursive self-improvement is possible.

      Any agent trying to improve itself has to use its existing intelligence and resources to do so.

      In this sense, humans, although unable to directly improve their source code, can be used as an example to a limited extent.

      Humanity can be seen as the parallel computation of a lot of copies of a certain artificial general intelligence.

      Yet there are no examples in which humans were able to rapidly and vastly make improvements that had the potential to yield further synergistic effects.

      In other words, it is true that human intelligence is largely fixed. So no recursive magic is possible there. But that doesn’t apply to our software. Yet there was no chess singularity, it took many decades. And we are still not at the point where computers can beat the best human players at the game of Go. Which seems to imply that our level of intelligence isn’t even able to quickly tweak a simple minded game AI to the point of reaching a level above our own. How then would we uplift our general intelligence given that we were handed our own source code? It doesn’t seem at all likely that something that could be called a “singularity” would be the result.

      But this is only a very small part of the argument and evidence on how recursive self-improvement is unlikely to work out in practice. There is a lot more here, if you follow the links (including an interview series with AI researchers, most of which believe it to be nearly impossible).

      I don’t doubt that the probability that general intelligence can be amplified due to speed improvements of different substrates is higher than that the brain is somehow a nearly optimal computing device.

      But that doesn’t seem to be enough evidence for a possibility of a singularity to occur this century.

      Here are the various possibilities:

      1.) Throw a lot of computing resources onto a problem by using parallel computing
      2.) Throwing more serial computing power at problems that can’t be parallelized
      3.) Come up with better algorithms

      First of all, I don’t see that parallel computing does yield the necessary advantage to spawn something that deserves the name “singularity”. As I wrote above, humanity can be seen as a massive parallelization of a static general intelligence. Yet it didn’t manage to come up with even a superhuman chess computer quickly.

      Another example. The U.S. has many more and smarter people than the Taliban. The bottom line was that the U.S. devoted a lot more output per man-hour to defeat a completely inferior enemy. Yet they basically lost.

      And regarding serial computing. The problem is that you won’t beat a human at Tic-tac-toe just because you thought about it for a million years.

      You also won’t get a practical advantage by throwing more computational resources at the travelling salesman problem and other problems in the same class.

      You are also not going to improve a conversation in your favor by improving each sentence for thousands of years. You will shortly hit diminishing returns. Especially since you lack the data to make accurate predictions. So obtaining empirical data does constitute a major bottleneck.

      So what about improvements to algorithms? I wrote about that too, if you follow the link above (conclusion: there seems to be no reason to believe that better than human general intelligent algorithms will be discovered any time soon).

      • DSimon

        Which seems to imply that our level of intelligence isn’t even able to quickly tweak a simple minded game AI to the point of reaching a level above our own. How then would we uplift our general intelligence given that we were handed our own source code? It doesn’t seem at all likely that something that could be called a “singularity” would be the result.

        I don’t think you’re giving us enough credit! On an evolutionary timescale, human civilization is an example of a highly parallel intelligence singularity.

        If we treat chess as the target problem: Evolution took millions of to develop human intelligence up to chess-capable levels, and human intelligence took thousands of years to form a civilization smart enough to properly learn and study chess.

        Compared to that, a few decades to build an even better chess-playing system than ourselves from scratch is not so bad. And it would’ve even faster if those researchers had started out with modern computing hardware.

        • http://kruel.co Alexander Kruel

          I was talking about a “singularity” in the context of AI risks. On an evolutionary timescale, or even cultural timescale, you are correct. Yet I do not believe that an AI would pose an existential risk if it would need a decade or more to amplify something as narrow as its chess intelligence.

          The issue is much more complicated though.

          And even if I disregard all the problems with rapid and vast recursive self-improvement, if I accept that it is 1.) logically possible 2.) physically possible 3.) economically feasible 4.) rational 5.) possible to acquire the necessary resources etc., then there are still many other issues that need to be true in order for it to make sense to contribute money to the SIAI.

          What else needs to be true: 6.) The fast computation of a simple algorithm is sufficient to outsmart and overpower humanity. 7.) Dangerous recursive self-improvement is the default outcome of the creation of artificial general intelligence. 7.) The human development of artificial general intelligence will take place quickly. 8.) The concept of friendly AI makes sense 9.) SIAI can solve friendly AI. 10.) SIAI does not increase risks from AI. 11.) SIAI does not increase negative utility by getting “friendliness” wrong. 12.) It makes sense to support SIAI at this time. 13.) Other problems or existential risks do not need to be faced before risks associated with artificial intelligence take place.

          I argue against all of those points in ‘Risks from AI and Charitable Giving’.

          To me it looks like that the case in favor of a donation to the Singularity Institute is highly conjunctive.

          • DSimon

            I do not think the AGI would take a decade. I brought up the larger timescales to point out that parallelization and relatively small implementation improvements (specialization, science) resulted in speed improvement of many orders of magnitude for humans, and therefore it’s not unreasonable to consider these approaches useful. They may be tapped out, but I don’t see any reason to believe they must be.

            Agreed on the specific weaknesses of SIAI. They need to do more to demonstrate they’re on to a useful strategy.

      • DSimon

        However, I think your point on an upper speed limit for gathering empirical data is a good one. It is certainly the case that in many practical situations, there is only so much you can do with simulations. Parallelization may be of some help, however: if you can run more experiments simultaneously, you can gather data more rapidly.

        • http://kruel.co Alexander Kruel

          What you have to consider with respect to large scale empirical experimentation is the following:

          We are already at a point where we have to build billion dollar chip manufacturing facilities to run our mobile phones. We need to build huge particle accelerators to obtain new insights into the nature of reality.

          An AI would either have to rely on the help of a whole technological civilization or be in control of advanced nanotech assemblers.

          And if an AI was to acquire the necessary resources on its own, its plan for world-domination would have to go unnoticed. This would require the workings of the AI to be opaque to its creators yet comprehensible to itself.

          (I can expand on this as well. It is much more complicated than it looks like. It is really hard to take over the world. Just consider the following minor point:

          Exploits only work for some systems. If you are dealing with different systems you will need different exploits. How do you reckon that such attacks won’t be visible and traceable? Packets do have to come from somewhere.

          And don’t forget that out systems become ever more secure and our toolbox to detect unauthorized use of information systems is becoming more advanced.)

  • http://twitter.com/blamer @blamer

    Is SI claiming the threat to homosapiens (iff singularity) will be from tech or transhumans? earthlings or offworld? or yet to be determined?

    • http://www.facebook.com/chris.hallquist Chris Hallquist

      The big ones they’re worried about are de novo AI and to a lesser extent uploads going crazy, and the latter are a kind of transhuman. Though they also argue that “Friendly AI” could mitigate other risks, so in that sense they’re concerned about those.

  • Pingback: yellow october()

  • Pingback: cat 4 brother()

  • Pingback: blue ofica()