Reading Yudkowsky, part 16

by Luke Muehlhauser on February 20, 2011 in Eliezer Yudkowsky,Resources,Reviews

AI researcher Eliezer Yudkowsky is something of an expert at human rationality, and at teaching it to others. His hundreds of posts at Overcoming Bias (now moved to Less Wrong) are a treasure trove for those who want to improve their own rationality. As such, I’m reading all of them, chronologically.

I suspect some of my readers want to improve their rationality, too. So I’m keeping a diary of my Yudkowsky reading. Feel free to follow along.

His 102nd post explains the Planning Fallacy, which helps explain why so many construction projects go overbudget. Also see Kahneman’s Planning Anecdote.

Conjunction Fallacy explains Kahneman’s and Tversky’s famous fallacy from 1982, with expanded discussion in Conjunction Controversy (or How They Nail it Down) and Burdensome Details.

Post #107 asks a very common question: What is Evidence?

Walking along the street, your shoelaces come untied.  Shortly thereafter, for some odd reason, you start believingyour shoelaces are untied.  Light leaves the Sun and strikes your shoelaces and bounces off; some photons enter the pupils of your eyes and strike your retina; the energy of the photons triggers neural impulses; the neural impulses are transmitted to the visual-processing areas of the brain; and there the optical information is processed and reconstructed into a 3D model that is recognized as an untied shoelace.  There is a sequence of events, a chain of cause and effect, within the world and your brain, by which you end up believing what you believe.  The final outcome of the process is a state of mind which mirrors the state of your actual shoelaces.

What is evidence? It is an event entangled, by links of cause and effect, with whatever you want to know about.  If the target of your inquiry is your shoelaces, for example, then the light entering your pupils is evidence entangled with your shoelaces.  This should not be confused with the technical sense of “entanglement” used in physics – here I’m just talking about “entanglement” in the sense of two things that end up in correlated states because of the links of cause and effect between them.

Not every influence creates the kind of “entanglement” required for evidence.  It’s no help to have a machine that beeps when you enter winning lottery numbers, if the machine also beeps when you enter losing lottery numbers.  The light reflected from your shoes would not be useful evidence about your shoelaces, if the photons ended up in the same physical state whether your shoelaces were tied or untied.

This is continued in The Lens That Sees Its Flaws:

Light leaves the Sun and strikes your shoelaces and bounces off; some photons enter the pupils of your eyes and strike your retina; the energy of the photons triggers neural impulses; the neural impulses are transmitted to the visual-processing areas of the brain; and there the optical information is processed and reconstructed into a 3D model that is recognized as an untied shoelace; and so you believe that your shoelaces are untied.

Here is the secret of deliberate rationality - this whole entanglement process is not magic, and you can understandit.  You can understand how you see your shoelaces.  You can think about which sort of thinking processes will create beliefs which mirror reality, and which thinking processes will not.

Mice can see, but they can’t understand seeing.  You can understand seeing, and because of that, you can do things which mice cannot do.  Take a moment to marvel at this, for it is indeed marvelous.

…The brain is a flawed lens through which to see reality.  This is true of both mouse brains and human brains.  But a human brain is a flawed lens that can understand its own flaws – its systematic errors, its biases – and apply second-order corrections to them.  This, in practice, makes the flawed lens far more powerful.  Not perfect, but far more powerful.

But now, How Much Evidence Does It Take? (The answer is highly mathematical.)

Applying this logic, Yudkowsky argues (in Einstein’s Arrogance) that Einstein may have been right to be so confident his theory was correct even before Eddington provided the first startling confirmation.

All this leads to a discussion of Occam’s Razor. “What does it mean for a theory to be complex?” is an old and difficult question, but a decent answer comes from Solomonoff induction.

9/26 is Petrov Day celebrates the man who averted World War III.

Previous post:

Next post:

{ 2 comments… read them below or add one }

MarkD February 20, 2011 at 10:47 pm

RE: Planning Fallacy: A good initial heuristic is to multiply the unknown party estimate by 2 and to multiply the known party by their performance. An altogether better methodology for project work is to adopt an agile framework and set better mid-term goals. Waterfall models always blur into cataracts (TM). Example: we will not fully automate the Denver baggage system but will automate the transfer of luggage from checkin to tarmac. We will then gather our lessons learned and revise estimates for next phase, etc.

RE: Solomonoff. There is a blurring of the history in Yudkowski. The groundwork was Kolmogorov with independent rediscovery by Greg Chaitin. Minimum Message Length (MML) and Minimum Description Length (MDL) are applications in the communications space, while Solomonoff is related to generalized induction. Related concepts emerge from compression theory (see Prediction by Partial Match, for instance). Also see AIC, BIC, and Sober’s 1975 Simplicity.

  (Quote)

piero February 21, 2011 at 10:11 pm

Re: Conjunction fallacy.

There is some mental process at work here that transforms the terms of the conjunction so that it cannot be so neatly coded. When I see the phrases “an accountant” and “an accountant who plays jazz” I picture two very different persons. In fact, the “accountant” bit in the second phrase has little in common with the first accountant. So I’d suggest that coding the phrases as X and X&Y is misleading; it’s more like X and X*&Y, where X* is the sort of accountant who would play jazz, not just any old accountant.

Part of the problem, I think, resides in the perceived incompatibility between accountancy and artistic flair. Consider which of these self-descriptions is more likely:
“I am an accountant, but my real passion is jazz”
“I am a jazz musician, but my real passion is accountancy”

I wonder what the results would be if the items to be ranked included the following:
A. Jeff is a sculptor
B. Jeff is a sculptor and a painter

  (Quote)

Leave a Comment