Reading Yudkowsky, part 29

by Luke Muehlhauser on April 15, 2011 in Eliezer Yudkowsky,Resources,Reviews

AI researcher Eliezer Yudkowsky is something of an expert at human rationality, and at teaching it to others. His hundreds of posts at Less Wrong are a treasure trove for those who want to improve their own rationality. As such, I’m reading all of them, chronologically.

I suspect some of my readers want to “level up” their rationality, too. So I’m keeping a diary of my Yudkowsky reading. Feel free to follow along.

Nine of his next 14 posts are collections of Rationality Quotes: 1, 2, 3, 4, 5, 6, 7, 8, 9. Here are some of my favorites:

In The Brothers Karamazov, Alyosha expresses the idea which panicked Dostoyevski more than any other: Without God, ‘everything is lawful’. But as Mohammed Atta can explain, the opposite is true. Without God, murder is forbidden by human law; it is only for those acting on behalf of God, that everything is permitted.

– Jonathan Wallace

Behind every story of extraordinary heroism, there is a less exciting and more interesting story about the larger failures that made heroism necessary in the first place.

– Black Belt Bayesian

I am most often irritated by those who attack the bishop but somehow fall for the securities analyst.

– Nassim Taleb, The Black Swan


How many legs does a dog have, if you call the tail a leg? Four. Calling a tail a leg doesn’t make it a leg.”

– Abraham Lincoln

Philosophy is the art of asking the wrong questions.

– J.R. Molloy

Beware `we should…’, extend a hand to ‘how do I…’

– Alan Cox

Things do get better, all the time, maybe just not as fast as I’d like. I do what I can. Don’t ask me to hate, too.

– Michael Wiik

The Allais Paradox describes “one of the first conflicts between decision theory and human reasoning to be experimentally exposed, in 1953.” Zut Allais! backs up and explains two key findings in the field of human heuristics and biases:

  • Experimental subjects tend to defend incoherent preferences even when they’re really silly.
  • People put very high values on small shifts in probability away from 0 or 1 (the certainty effect).

Allais Malaise makes a few more clarifications.

Against Discount Rates concludes with the following:

I’m not advocating against the idea that Treasury bonds can exist.  But I am advocating against the idea that you should intrinsically care less about the future than the present; and I am advocating against the idea that you should compound a 5% discount rate a century out when you are valuing global catastrophic risk management.

Circular Altruism opens with a dilemma:

Suppose that a disease, or a monster, or a war, or something, is killing people.  And suppose you only have enough resources to implement one of the following two options:

  1. Save 400 lives, with certainty.
  2. Save 500 lives, with 90% probability; save no lives, 10% probability.

Most people choose option 1.  Which, I think, is foolish; because if you multiply 500 lives by 90% probability, you get an expected value of 450 lives, which exceeds the 400-life value of option 1.  (Lives saved don’t diminish in marginal utility, so this is an appropriate calculation.)

“What!” you cry, incensed.  “How can you gamble with human lives? How can you think about numbers when so much is at stake?  What if that 10% probability strikes, and everyone dies?  So much for your damned logic!  You’re following your rationality off a cliff!”

Ah, but here’s the interesting thing.  If you present the options this way:

  1. 100 people die, with certainty.
  2. 90% chance no one dies; 10% chance 500 people die.

Then a majority choose option 2.  Even though it’s the same gamble. You see, just as a certainty of saving 400 lives seems to feel so much more comfortable than an unsure gain, so too, a certain loss feels worse than an uncertain one.

Previous post:

Next post:

{ 5 comments… read them below or add one }

Garren April 15, 2011 at 9:06 am

That’s such a neat pair of survey questions at the end. The tendency to far overvalue (perceived or stipulated) certainty is surely the greatest threat to rationality. It’s as if humans are not designed to treat 98% probability as closer to 100% than 80%.

  (Quote)

drew April 15, 2011 at 3:44 pm

*sigh* these reading through yudkowsky posts are getting really old and tiresome. Time to cut it out altogether.

  (Quote)

JoeK April 15, 2011 at 8:33 pm

The Wallace quote confused me for a minute. “Without God, ‘everything is lawful’.” Well, that’s true! thinks I. Everything works by natural law, no god needed :-)

  (Quote)

Martin April 15, 2011 at 8:37 pm

Without God, murder is forbidden by human law; . . .

Except when it isn’t, then it’s not murder. unless there is some greater truth that human law rests on

  (Quote)

Ben A April 16, 2011 at 9:02 am

Rephrasing the two choices in the 2nd manner is a little misleading because they lack context and definition.

First, we have to assume that’s there’s 500 people total for every option, even though the options don’t state that.

“100 people die, with certainty.” makes me think that at least 100 people must die, and that more could die in the future, which the option doesn’t state – we must assume.

  (Quote)

Leave a Comment