Reading Yudkowsky, part 34

by Luke Muehlhauser on May 5, 2011 in Eliezer Yudkowsky,Resources,Reviews

AI researcher Eliezer Yudkowsky is something of an expert at human rationality, and at teaching it to others. His hundreds of posts at Less Wrong are a treasure trove for those who want to improve their own rationality. As such, I’m reading all of them, chronologically.

I suspect some of my readers want to “level up” their rationality, too. So I’m keeping a diary of my Yudkowsky reading. Feel free to follow along.

His 270th post is Leave a Line of Retreat, which opens with two quotes:

When you surround the enemy
Always allow them an escape route.
They must see that there is
An alternative to death.

– Sun Tzu, The Art of War, Cloud Hands edition

Don’t raise the pressure, lower the wall.

– Lois McMaster Bujold, Komarr

Then, the story:

Last night I happened to be conversing with a nonrationalist who had somehow wandered into a local rationalists’ gathering.  She had just declared (a) her belief in souls and (b) that she didn’t believe in cryonics because she believed the soul wouldn’t stay with the frozen body.  I asked, “But how do you know that?”  From the confusion that flashed on her face, it was pretty clear that this question had never occurred to her.  I don’t say this in a bad way – she seemed like a nice person with absolutely no training in rationality, just like most of the rest of the human species.

…”Make sure,” I suggested to her, “that you visualize what the world would be like if there are no souls, and what you would do about that.  Don’t think about all the reasons that it can’t be that way, just accept it as a premise and then visualize the consequences.  So that you’ll think, ‘Well, if there are no souls, I can just sign up for cryonics’, or ‘If there is no God, I can just go on being moral anyway,’ rather than it being too horrifying to face.  As a matter of self-respect you should try to believe the truth no matter how uncomfortable it is, like I said before; but as a matter of human nature, it helps to make a belief less uncomfortable, before you try to evaluate the evidence for it.”

The point is that you have to recognize you’re a human being, so in order to avoid rejecting positions because you don’t like their consequences, you should find a way to be emotionally okay with whatever happens to be true:

As Sun Tzu advises you to do with your enemies, you must do with yourself – leave yourself a line of retreat, so that you will have less trouble retreating.  The prospect of losing your job, say, may seem a lot more scary when you can’t even bear to think about it, than after you have calculated exactly how long your savings will last, and checked the job market in your area, and otherwise planned out exactly what to do next.  Only then will you be ready to fairly assess the probability of keeping your job in the planned layoffs next month.  Be a true coward, and plan out your retreat in detail – visualize every step – preferably before you first come to the battlefield.

The hope is that it takes less courage to visualize an uncomfortable state of affairs as a thought experiment, than to consider how likely it is to be true.  But then after you do the former, it becomes easier to do the latter.

How many religious people would retain their belief in God, if they could accurately visualize that hypothetical world in which there was no God and they themselves have become atheists?

Leaving a line of retreat is a powerful technique, but it’s not easy.  Honest visualization doesn’t take as much effort as admitting outright that God doesn’t exist, but it does take an effort.

The Second Law of Thermodynamics, and Engines of Cognition explains the second law of thermodynamics in terms of Bayes’ Theorem, and then makes an analogy between thermodynamics and human cognition:

To form accurate beliefs about something, you really do have to observe it. It’s a very physical, very real process: any rational mind does “work” in the thermodynamic sense, not just the sense of mental effort…  So unless you can tell me which specific step in your argument violates the laws of physics by giving you true knowledge of the unseen, don’t expect me to believe that a big, elaborate clever argument can do it either.

Perpetual Motion Beliefs continues:

Each extra detail in your argument necessarily decreases the joint probability.  The probability that you’ve violated the Second Law of Thermodynamics without knowing exactly how, by guessing the exact state of boiling water without evidence, so that you can stick your finger in without getting burned, is, necessarily, even less than the probability of sticking in your finger into boiling water without getting burned.

I say all this, because people really do construct these huge edifices of argument in the course of believing without evidence.

Searching for Bayes-Structure gets even more abstract, and ends:

To see through the surface adhockery of a cognitive process, to the Bayesian structure underneath – to perceive the probability flows, and know how, not just know that, this cognition too is Bayesian – as it always is – as it always must be – to be able to sense the Force underlying all cognition – this, is the Bayes-Sight.

Conditional Independence and Naive Bayes continues that line of thought. Far easier to read is Words as Mental Paintbrush Handles:

Can you visualize a “green dog”?  Can you visualize a “cheese apple”?

“Apple” isn’t just a sequence of two syllables or five letters.  That’s a shadow.  That’s the tip of the tiger’s tail.

Words, or rather the concepts behind them, are paintbrushes – you can use them to draw images in your own mind.  Literally draw, if you employ concepts to make a picture in your visual cortex.  And by the use of shared labels, you can reach into someone else’s mind, and grasp their paintbrushes to draw pictures in their minds – sketch a little green dog in their visual cortex.

Or how would you be able to visualize combinations like a “triangular lightbulb” – imposing triangleness on lightbulbs, keeping the essence of both, even if you’ve never seen such a thing in your life?

Don’t make the mistake the behaviorists made.  There’s far more to speech than sound in air.  The labels are just pointers – “look in memory area 1387540″.  Sooner or later, when you’re handed a pointer, it comes time to dereference it, and actually look in memory area 1387540.

This is followed by two more sets of rationality quotes.

Previous post:

Next post:

{ 8 comments… read them below or add one }

cl May 5, 2011 at 9:57 pm

Last night I happened to be conversing with a nonrationalist who had somehow wandered into a local rationalists’ gathering. She had just declared (a) her belief in souls

I like much of what Yudkowsky writes, I really do, but this type of crap is pure, intellectually chauvinist smarm.


cl May 6, 2011 at 12:26 pm

Hey Luke here’s an unrelated suggestion that might facilitate conversation here. You know how most blogs have a little “recent comments” sidebar? I suggest you implement one of those so the rest of us can be informed when others comment on old threads. For example, I’m about to make a comment on an old thread, one where I concede that one of my arguments wasn’t ship-shape, and I think it would be nice if other people could see that. It just seems like having a “recent comments” sidebar would spur additional conversation, which might be a good idea since the whole “slowed posting” situation.

Your thoughts?


Ralph May 6, 2011 at 12:55 pm

Luke, what cl posted seems to have a good idea. Any difficulties in implementing it?


Ralph May 6, 2011 at 12:56 pm

hmph! “seems to be” not “seems to have”


Luke Muehlhauser May 6, 2011 at 1:14 pm

On recent comments, I just don’t like the clutter.


cl May 6, 2011 at 3:16 pm


So, your desire for clean aesthetics trumps our desire to obtain quality information in the pursuit of truth? Isn’t the desire for knowledge stronger and more important than the desire for clean aesthetics? Surely your approach to knowledge isn’t the same as SOME WOMEN’S approach to cars [i.e. I want the one that looks prettiest and I don't care how well it runs]? It seems to me that having this feature would satisfy more and stronger desires than it thwarts, and we could even conduct a poll that would serve as a sort of “preliminary evidence” — so that we don’t make the same mistake Alonzo Fyfe makes practically every Tuesday.

You are a desirist, right? If so, conduct the poll. Let’s put the theory to work. I’d say that “CSA commenters generally” have strong reason to promote this feature, and that you don’t have as strong a reason to condemn it. Prove me wrong.


Garren May 6, 2011 at 3:41 pm


Does this mean we can vote your sidebar quotes off the island? ;)


cl May 6, 2011 at 4:12 pm

What’s wrong with my sidebar quotes? Some of them are outright hilarious. In fact, they could use an update.


Leave a Comment