Reading Yudkowsky, part 2

by Luke Muehlhauser on November 22, 2010 in Eliezer Yudkowsky,Resources,Reviews

AI researcher Eliezer Yudkowsky is something of an expert at human rationality, and at teaching it to others. His hundreds of posts at Overcoming Bias (now moved to Less Wrong) are a treasure trove for those who want to improve their own rationality. As such, I’m reading all of them, chronologically.

I suspect some of my readers want to improve their rationality, too. So I’m keeping a diary of my Yudkowsky reading. Feel free to follow along.

Yudkowsky’s 7th post is A Fable of Science and Politics, which is a lovely and exciting parable about a race of people forced to live underground for centuries. There are legends of a thing called sky, an endless expanse in which tufs of cotton float. But the underground world is divided over whether this “sky” is blue or green. Blues and Greens vote differently, and for centuries were violent to one another. Then an explorer in the upper caverns stumbles upon an opening to the sky and discovers its true color and… well, you have to read the whole story. It’s worth it.

Next is Some Claims Are Just Too Extraordinary, which opens with this quote from Peter K. Bertine:

If a ship landed in my yard and LGMs stepped out, I’d push past their literature and try to find the cable that dropped the saucer on my roses. Lack of a cable or any significant burning to the flowers, I’d then grab a hammer and start knocking about in the ship till I was convinced that nothing said “Intel Inside.” Then when I discovered a “Flux Capacitor” type thing I would finally stop and say, “Hey, cool gadget!” Assuming the universal benevolence of the LGMs, I’d yank it out and demand from the nearest “Grey” (they are the tall nice ones), “where the hell did this come from?” Greys don’t talk, they communicate via telepathy, so I’d ignore the voice inside my head. Then stepping outside the saucer and sitting in a lawn chair, I’d throw pebbles at the aliens till I was sure they were solid. Then I’d look down at the “Flux Capacitor” and make sure it hadn’t morphed into my bird feeder. Finally, with proof in my hand and aliens sitting on my deck (they’d be offered beers, though I’ve heard that they absorb energy like a plant) I’d grab my cell phone and tell my doctor that I’m having a serious manic episode with full-blown visual hallucinations.

Yudkowsky offers just as startling a conclusion:

We underestimate the power of science, and overestimate the power of personal observation. A peer-reviewed, journal-published, replicated report is worth far more than what you see with your own eyes. Our own eyes can deceive us. People can fool themselves, hallucinate, and even go insane. The controls on publication in major journals are more trustworthy than the very fabric of your brain. If you see with your own eyes that the sky is blue, and Science says it is green, then sir, I advise that you trust in Science.

Unfortunately for this post, Yudkowky only makes time for assertion, not argument. But I probably agree with him already, which may explain my debate with Richard Carrier.

In his next post, Yudkowsky discusses the claim:

Outside the laboratory, scientists are no wiser than anyone else.

He thinks it’s not exactly true. There’s probably some tiny correlation between being a scientist and being wise. But there doesn’t seem to be much. And Yudkowsky says this lack of correlation should cause us to “Sit bolt upright in alarm.”

Suppose we discover that a Ph.D. economist buys a lottery ticket every week. We have to ask ourselves: Does this person really understand expected utility, on a gut level?


Now what are we to think of a scientist who seems competent inside the laboratory, but who, outside the laboratory, believes in a spirit world? We ask why, and the scientist says something along the lines of: “Well, no one really knows, and I admit that I don’t have any evidence – it’s a religious belief, it can’t be disproven one way or another by observation.” I cannot but conclude that this person literally doesn’t know why you have to look at things. They may have been taught a certain ritual of experimentation, but they don’t understand the reason for it – that to map a territory, you have to look at it – that to gain information about the environment, you have to undergo a causal process whereby you interact with the environment and end up correlated to it…

Maybe our spiritual scientist says: “But it’s not a matter for experiment. The spirits spoke to me in my heart.” Well, if we really suppose that spirits are speaking in any fashion whatsoever, that is a causal interaction and it counts as an observation. Probability theory still applies.

In Politics is the Mind-Killer, Yudkowsky makes a simple plea to avoid slipping in politics stabs when discussing rationality. We naked apes have enough trouble being rational without having our political emotions pricked.

In Just Lose Hope Already, Yudkowsky notes one way rationality can be useful: it can help you avoid really stupid moves, such as not turning little mistakes into huge mistakes.

In You Are Not Hiring the Top 1%, Yudkowsky points out the statistical mistake being made when companies think that by selecting the top candidate out of 100 applications, they are therefore hiring the top 1%.

In Policy Debates Should Not Be One-Sided, Yudkowsky uses Robin Hanson’s proposal of a “banned products shop” to argue that while the evidence for matters of fact (like biological evolution) can often be one-sided, this need not be the case for policies, which often have surprising, double-sided effects.

Next, Yudkowsky names Burch’s Law:

If people have a right to be stupid, the market will respond by supplying all the stupidity that can be sold.

Next is The Scales of Justice, the Notebook of Rationality, where Yudkowsky highlights a specific kind of error we make. We tend to judge things – for example, a technology – with either a positive or negative attitude. We add up the positive points and negative points and we have our result.

But in the real world, all the facts we use in the debate are logically distinct. The facts don’t know which “side” they’re on.

This leads to false judgments in the real world:

If you tell people a reactor design produces less waste, they rate its probability of meltdown as lower. This means getting the wrong answer to physical questions with definite factual answers, because you have mixed up logically distinct questions…

In Blue or Green on Regulation, Yudkowsky complains that often when he makes a simple factual prediction, he is interpreted as defending some broad policy (probably, the one usually being endorsed when that particular fact is trotted out). This has happened to me, too. I’ve probably done it to others.

But really, the post is a great back-and-forth about government regulation and anti-regulation forces, and how that interaction occurs. A very insightful analysis.

In Superstimuli and the Collapse of Western Civilization, Yudkowsky notes that people are now dying – or losing jobs and spouses – because videogames are so stimulating that some people would rather play consecutive hour #57 than eat or sleep. That’s powerful stuff.

The candy bar is a superstimulous: a collection of fat, sugar, and salt far denser than anything we encountered in our ancestral environment.

So is the Photoshopped model on the magazine cover. No live, physical woman on the planet has ever pushed that many buttons in the male sex drive so perfectly. Only hours of makeup and lighting and Photoshop can do that.

The last two paragraphs fulfill Yudkowsky’s title:

I leave you with a final argument from fictional evidence: Simon Funk’s online novel After Life depicts (among other plot points) the planned extermination of biological Homo sapiens – not by marching robot armies, but by artificial children that are much cuter and sweeter and more fun to raise than real children. Perhaps the demographic collapse of advanced societies happens because the market supplies ever-more-tempting alternatives to having children, while the attractiveness of changing diapers remains constant over time. Where are the advertising billboards that say “BREED”? Who will pay professional image consultants to make arguing with sullen teenagers seem more alluring than a vacation in Tahiti?

“In the end,” Simon Funk wrote, “the human species was simply marketed out of existence.”

Previous post:

Next post:

{ 8 comments… read them below or add one }

Brian November 22, 2010 at 11:21 am

When I was in Rabbinical School, one of the Deans told the students that he bought one lottery ticket every week. It wasn’t an economic investment (it isn’t unless you have crushing debt and will lose your small amount of money anyway), but taking advantage of human nature. People think they will win, and this creates energizing and (potentially) useful feelings. It obviously works even when we know the probabilities in question.

By extension, having more than one outstanding lottery ticket is useless or worse. However, this doesn’t mean that buying a lottery ticket is an irrational act.


ildi November 22, 2010 at 11:50 am

artificial children that are much cuter and sweeter and more fun to raise than real children

(They’re called pets.)


Scott Scheule November 22, 2010 at 1:45 pm

Whatever we know as rationality starts to break down at this point. If one possesses irrationalities, and if one knows he possesses such irrationalities, is altering one’s actions to take those irrationalities into account more or less rational? In some sense, you’re protecting those irrationalities and refusing to simply eliminate the irrationalities entirely–in another sense, you’re just being realistic.

I like smoking cigarettes. I know if I get near one, I’ll have a desire to smoke it, though rationally I’d prefer not to smoke it. Is it thus rational to avoid being near cigarettes? Many would, I imagine, say yes.

This is the same situation as the scientist buying the lottery ticket. Logically he may know that the odds of the lottery ticket winning are ridiculous slight, just as I know that, logically, it would be better not to smoke. But I’ve got an urge to smoke, and he gets a high from gambling–we take these into account, presumably, because they are stubborn irrationalities we can’t uproot.

We’re oranges, not clockwork.


rupert mirdock November 22, 2010 at 10:00 pm

off topic to this article, but relevant to website, I’m really looking forward to your reviews of the dembski-hitchens debate (just started watching) and the shermer-ridley-dawkins v. craig-wolpe TED-style debate (just finished watching in English a few days ago).


Jeff H November 23, 2010 at 12:27 pm

The irrationality of something like buying lottery tickets depends on the motivation. If someone buys them expecting to win, that’s clearly irrational. But if someone buys them for the thrill of it, or for the the sheer enjoyment, or whatever other reason, that can be perfectly okay. In that case, it’s an arational choice as far as I’m concerned. Spending money to enjoy lottery tickets is as irrational as spending money to enjoy building model airplanes, or to watch sporting events, etc. As long as it’s done in moderation, I don’t think rationality really comes into play.


Scott Scheule November 23, 2010 at 12:34 pm

Jeff, but note in the comments that Yudkowsky explicitly argues against the “thrill of it” option.

“…if a PhD economist has pleasurable dreams about winning the lottery, that is exactly what I would call “failing to understand probability on a gut level”. Look at the water! A calculated probability of 0.0000001 should diminish the emotional strength of any anticipation, positive or negative, by a factor of ten million. Otherwise you’ve understood the probability as little symbols on paper but not what it *means* in real life.”

In his eyes, the fact that you get a thrill is symptomatic of your underlying irrationality in believing you have a chance. The rational actor would not, according to Eliezer, get a thrill.

You might respond–I would–who the hell are you to tell me what’s rational to be thrilled about? All desires are arbitrary! Why does it make any less sense to get a thrill from a lottery ticket rather than abstract art, or food, or procreation? I’d agree.


Jeff H November 23, 2010 at 12:35 pm

On another note, sorry to toot my own horn here, Luke, but I wanted to mention that I’ve started a series on my blog called Contesting Christianity. It is going to go through what I feel are the most common arguments that Christians use to support their religion – but I am going to come at it from a layman’s perspective, so it’s going to be fairly “basic”. I’m not assuming much prior knowledge.

Just thought people might be interested. Send the link to that annoying Christian on Facebook who always quotes Bible verses on their status. It’ll do ‘em some good. :)

/end advertisement


Layne November 23, 2010 at 7:58 pm

Am I just not seeing part of “A Fable of Science and Politics” or does it end with Ferris exploring? That seemed abrupt.


Leave a Comment