At LessWrong: Why a human (or group of humans) might create unFriendly AI halfway on purpose

This post that I just wrote contains some of the stuff on evolutionary psychology I’ve been meaning to write more about here:

Too many people–at least, too many writers of the kind of fiction where the villain turns out to be an all-right guy in the end–seem to believe that if someone is the hero of their own story and genuinely believes they’re doing the right thing, they can’t really be evil. But you know who was the hero of his own story and genuinely believed he was doing the right thing? Hitler. He believed he was saving the world from the Jews and promoting the greatness of the German volk.

We have every reason to think that the psychological tendencies that created these hero-villains are nearly universal. Evolution has no way to give us nice impulses for the sake of having nice impulses. Theory predicts, and observation confirms, that we tend to care more about blood-relatives than mere allies and allies more than strangers. As Hume observed (remarkably, without any knowledge of Hammilton’s rule) “A man naturally loves his children better than his nephews, his nephews better than his cousins, his cousins better than strangers, where every thing else is equal.” And we care more about ourselves than any single other individual on the planet (even if we might sacrifice ourselves for two brothers or eight cousins.)

Most of us are not murderers, but then most of have never been in a situation where it would be in our interest to commit murder. The really disturbing thing is that there is much evidence that ordinary people can become monsters as soon as the situation changes. Science gives us the Stanford Prison Experiment and Milgram’s experiment on obedience to authority, history gives us even more disturbing facts about how many soldiers commit atrocities in war time. Of the soldiers who came from societies where atrocities are frowned on, most of them must have seemed perfectly normal before they went off to war. Probably most of them, if they’d thought about it, would have sincerely believed they were incapable of doing such things.

This makes a frightening amount of evolutionary sense. There’s reason for evolution to, as much as possible, give us conditional rules for behavior so we only do certain things when it’s fitness increasing to do so. Normally, doing the kind of things done during the Rape of Nanking leads to swift punishments, but the circumstances when such things actually happen tend to be circumstances where punishment is much less likely, where the other guys are trying to kill you anyway and your superior officer is willing to at minimum look the other way. But if you’re in a situation where doing such things is not in your interest, where’s the evolutionary benefit of even being aware of what you’re capable of?

"Atomsk - Yes, I think the way I feel about it is normal. I think ..."

Let’s talk about violent pornography
"The Scientific Method works by testing a hypothesis for implications, contradictions, and ridiculous/false results. You ..."

Pulling some devastating punches: a review ..."
"A bit OT: Found this article and it is imo closely related to the issue ..."

Let’s talk about violent pornography
"Just one thing for now, because it takes quite a bit of time to think ..."

Let’s talk about violent pornography

Browse Our Archives

What Are Your Thoughts?leave a comment