In this comment thread, there was one point where I ended up just quoting Eliezer Yudkowsky at length, and more points where I was tempted to do so. But I think it would be more productive to write a post telling people what papers they should read if they have a serious interest in this stuff. Here’s my list, which is similar but not identical to lists that other people like Luke Muehlhauser might give:
- Eliezer Yudkowsky. 2008. Artificial Intelligence as a Positive and Negative Factor in Global Risk. I don’t agree with Eliezer on everything, but at the very least if you’re going to criticize his views you should read the main academic paper where he lays them out.
- Luke Muehlhauser and Anna Salamon. 2012. Intelligence Explosion: Evidence and Import. One thing that really impresses me about Luke is that he seems to have mastered the art of identifying claims that are relatively non-contentious but also very important if true. I agree with virtually everything in this paper, aside from a few nitpicks. One of the best parts of this paper is the discussion of the advantages digital minds would have over meat minds.
- Kaj Sotala. 2012. Advantages of Artificial Intelligences, Uploads, and Digital Minds. The title is self-explanatory. I’m including a second paper with this title just because I think the issue is so important.
- Nick Bostrom. 2012. The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents. A more rigorous explanation of why a superintelligent AI is potentially dangerous. TLDR; intelligence doesn’t imply benevolence, and it may not imply malevolence as an end goal, but it does imply certain instrumental goals unless they would conflict with its other goals.
- Anders Sandberg and Nick Bostrom. 2008. Whole Brain Emulation: A Roadmap. This is the thing that originally made me go “oh crap, this technology isn’t just theoretically possible, it could happen this century.”