SMBC on intelligence explosion

The webcomic Saturday Morning Breakfast Cereal has a new comic up on the idea of intelligence explosion. Go read it.

The portrayal of the scientist and the AI in the comic is stereotyped, of course, but that’s really beside the point. Ignoring the irrelevant stereotyping, we can nitpick the plausibility a bit: unless you program it to do so, an AI isn’t going to care about getting “displaced” as an ultimate goal. I recommend reading Nick Bostrom here; the key point is there aren’t any ultimate goals that come automatically with intelligence. And it’s really unclear what the goals of the AI in the comic are supposed to be.

At the same time, as Bostrom also points out, all else being equal an AI should want to stay around so it can continue carrying out whatever goals it does have. But if it can design a better AI with those very same goals, displacement isn’t so much a worry. There’s also the possibility of self-modifying to be smarter.

Nitpicking aside, it’s nice to see a popular webcomic like SMBC giving the idea of intelligence explosion–and the possible dangers associated with it–some coverage.