Reading Yudkowsky, part 7

by Luke Muehlhauser on December 16, 2010 in Eliezer Yudkowsky,Resources,Reviews

AI researcher Eliezer Yudkowsky is something of an expert at human rationality, and at teaching it to others. His hundreds of posts at Overcoming Bias (now moved to Less Wrong) are a treasure trove for those who want to improve their own rationality. As such, I’m reading all of them, chronologically.

I suspect some of my readers want to improve their rationality, too. So I’m keeping a diary of my Yudkowsky reading. Feel free to follow along.

Yudkowsky’s 39th post is Feeling Rational:

A popular belief about “rationality” is that rationality opposes all emotion – that all our sadness and all our joy are automatically anti-logical by virtue of being feelings. Yet strangely enough, I can’t find any theorem of probability theory which proves that I should appear ice-cold and expressionless.

So is rationality orthogonal to feeling? No; our emotions arise from our models of reality. If I believe that my dead brother has been discovered alive, I will be happy; if I wake up and realize it was a dream, I will be sad…

In my early days I was never quite certain whether it was all right to feel things strongly – whether it was allowed, whether it was proper… Since the days of Socrates at least, and probably long before, the way to appear cultured and sophisticated has been to never let anyone see you care strongly about anything…

But I know, now, that there’s nothing wrong with feeling strongly. Ever since I adopted the rule of “That which can be destroyed by the truth should be,” I’ve also come to realize “That which the truth nourishes should thrive.” When something good happens, I am happy, and there is no confusion in my mind about whether it is rational for me to be happy. When something terrible happens, I do not flee my sadness by searching for fake consolations and false silver linings.

In Universal Fire, Yudkowsky discusses a fictional story where matches do not light. He explains that the same features of the universe that light a match also allow us to breathe. If matches didn’t light, we couldn’t breathe. “Reality is laced together a lot more tightly than humans might like to believe.”

In Universal Law, Yudkowsky remarks that:

It wasn’t just that Newton had dared to unify the Earthly realm of base matter with the obviously different and sacred celestial realm, once thought to be the abode of the gods. Newton’s discovery gave rise to the notion of a universal law, one that is the same everywhere and everywhen, with literally zero exceptions…

If you would learn to think like reality, then here is the Tao:

Since the beginning
not one unusual thing
has ever happened.

Think Like Reality extends this thought:

Whenever I hear someone describe quantum physics as “weird” – whenever I hear someone bewailing the mysterious effects of observation on the observed, or the bizarre existence of nonlocal correlations, or the incredible impossibility of knowing position and momentum at the same time – then I think to myself:  This person will never understand physics no matter how many books they read.

Reality has been around since long before you showed up.  Don’t go calling it nasty names like “bizarre” or “incredible”.  The universe was propagating complex amplitudes through configuration space for ten billion years before life ever emerged on Earth.  Quantum physics is not “weird”.  You are weird.  You have the absolutely bizarre idea that reality ought to consist of little billiard balls bopping around, when in fact reality is a perfectly normal cloud of complex amplitude in configuration space.  This is your problem, not reality’s, and you are the one who needs to change…

[But] it is pointless to pretend that quantum physics feels natural to you when in fact it feels strange. This is merely denying your confusion, not becoming less confused. But it will also hinder you to keep thinking How bizarre! Spending emotional energy on incredulity wastes time you could be using to update. It repeatedly throws you back into the frame of the old, wrong viewpoint. It feeds your sense of righteous indignation at reality daring to contradict you.

Of course, that doesn’t mean you shouldn’t admit to being surprised by reality. In fact, says Yudkowsky, you should Beware the Unsurprised.

The Third Alternative opens this way:

“Believing in Santa Claus gives children a sense of wonder and encourages them to behave well in hope of receiving presents. If Santa-belief is destroyed by truth, the children will lose their sense of wonder and stop behaving nicely. Therefore, even though Santa-belief is false-to-fact, it is a Noble Lie whose net benefit should be preserved for utilitarian reasons.”

Classically, this is known as a false dilemma, the fallacy of the excluded middle, or the package-deal fallacy. Even if we accept the underlying factual and moral premises of the above argument, it does not carry through. Even supposing that the Santa policy (encourage children to believe in Santa Claus) is better than the null policy (do nothing), it does not follow that Santa-ism is the best of all possible alternatives. Other policies could also supply children with a sense of wonder, such as taking them to watch a Space Shuttle launch or supplying them with science fiction novels. Likewise (if I recall correctly), offering children bribes for good behavior encourages the children to behave well only when adults are watching, while praise without bribes leads to unconditional good behavior.

Noble Lies are generally package-deal fallacies; and the response to a package-deal fallacy is that if we really need the supposed gain, we can construct a Third Alternative for getting it.

Can we have an example? How about Third Alternative for Afterlife-ism?

One of the most commonly proposed Noble Lies is belief in an afterlife.  Surely, goes the argument, the crushing certainty of absolute annihilation in a few decades is too much for any human being to bear.  People need hope – if they don’t believe in an afterlife, they won’t be able to live.

Surely this must be the strongest of all arguments for Noble Lies.  You can find Third Alternatives to many dilemmas, but can you find one to Death?

Well, did you close your eyes and think creatively about the problem for five minutes? …

The assumed task is to find a source of hope against looming death.  So at the very least I would cite medical nanotechnology, the argument from actuarial escape velocitycryonics, or meddling with the forbidden ultimate technology.  But do you think that anyone who actually argued for afterlife as a Noble Lie would be glad to hear about these Third Alternatives?  No, because the point was not really to find the best strategy for supplying hope, but rather to excuse a fixed previous belief from criticism.

You can argue against the feasibility of one of the above Third Alternatives, or even argue against the feasibility of all of them, but that’s not the point.  Any one of those Third Alternatives stretches credulity less than a soul – that is (a) an imperishable dualistic stuff floating alongside the brain which (b) malfunctions exactly as the brain is neurologically damaged and yet (c) survives the brain’s entire death. Even if we suppose the above Third Alternatives to be false-in-fact, they are packaged with far fewer associated absurdities, and put far less of a strain on the Standard Model.

Previous post:

Next post:

{ 2 comments… read them below or add one }

Adito December 16, 2010 at 12:04 pm

I agree with most of that but I’m not sure I buy his Santa story. A child will always be filled with false beliefs held for utility. It would be impossible to stomp them all out and replace them with real things that give them a sense of wonder. A childs mind naturally explores its own boundaries and eventually learns to tell truth from fiction. We can help the process but I think it’s a mistake to try and dominate it.

  (Quote)

Leon December 17, 2010 at 5:30 pm

If you would learn to think like reality, then here is the Tao:
Since the beginning
not one unusual thing
has ever happened.

I agree with the sentiment — that “unusual” only makes sense with respect to “flawed” human expectations — but what does Yudkowsky mean by “reality” here? In other places, he repeatedly uses the “map/territory” analogy in a way which seems dodgy to me, with “territory” referring not to “reality” or raw experience, but to the most fine-grained, low-level model we have available. To me, that’s more like “our best map”, not “the territory”, which in some sense we can’t really talk about, since talking is mapmaking. The above quote seems to refer more to an ideal map/theory than to the territory/reality — to thinking, in other words, like a perfectly endowed cartographer (maybe God), rather than one who, as an embodied being, is actually within reality, trying to build a map from the mess of experience that presents itself to us.

I guess I’m wondering what the point is of trying to view things from an unattainable vantage point.

Any one of those Third Alternatives stretches credulity less than a soul – that is (a) an imperishable dualistic stuff floating alongside the brain which (b) malfunctions exactly as the brain is neurologically damaged and yet (c) survives the brain’s entire death. Even if we suppose the above Third Alternatives to be false-in-fact, they are packaged with far fewer associated absurdities, and put far less of a strain on the Standard Model.

The Christian alternative is not about disembodied souls. Although there is some kind of continuity implied, the language used is of re-creation and re-birth. See 2 Corinthians 5 (which explicitly denies [the hope for] an immaterial future state), the resurrection of Jesus, the first few chapters of Genesis, or the Old Testament prophets. I also doubt that the use of “soul” as analogous to consciousness (or some kind of homunculus) is in line with the New Testament. Perhaps this is only a small improvement credibility-wise, but at least we’re out of Hallmark territory.

  (Quote)

Leave a Comment