Reading Yudkowsky, part 6

by Luke Muehlhauser on December 8, 2010 in Eliezer Yudkowsky,Resources,Reviews

AI researcher Eliezer Yudkowsky is something of an expert at human rationality, and at teaching it to others. His hundreds of posts at Overcoming Bias (now moved to Less Wrong) are a treasure trove for those who want to improve their own rationality. As such, I’m reading all of them, chronologically.

I suspect some of my readers want to improve their rationality, too. So I’m keeping a diary of my Yudkowsky reading. Feel free to follow along.

Today I jump out of sequence to consider one of Yudkowsky’s central essays, The Simple Truth, which begins:

This essay is meant to restore a naive view of truth.

The essay is really a parable. Like my other “Reading Yudkowsky” posts, I wanted to summarize Yudkowsky’s essay and quote a few passages, but I found it was easier to rewrite the parable in shorter form, where I play the character of the shepherd instead of Eliezer. But then that failed, too. I really can’t improve on Yudkowsky original parable. So just go read it.

Another classic is An Intuitive Explanation of Bayes’ Theorem, which also cannot be summarized but must be read in full. But here is the beginning:

Your friends and colleagues are talking about something called “Bayes’ Theorem” or “Bayes’ Rule”, or something called Bayesian reasoning. They sound really enthusiastic about it, too, so you google and find a webpage about Bayes’ Theorem and…

It’s this equation. That’s all. Just one equation. The page you found gives a definition of it, but it doesn’t say what it is, or why it’s useful, or why your friends would be interested in it. It looks like this random statistics thing.

So you came here. Maybe you don’t understand what the equation says. Maybe you understand it in theory, but every time you try to apply it in practice you get mixed up trying to remember the difference between p(a|x) and p(x|a), and whether p(a)*p(x|a) belongs in the numerator or the denominator. Maybe you see the theorem, and you understand the theorem, and you can use the theorem, but you can’t understand why your friends and/or research colleagues seem to think it’s the secret of the universe. Maybe your friends are all wearing Bayes’ Theorem T-shirts, and you’re feeling left out. Maybe you’re a girl looking for a boyfriend, but the boy you’re interested in refuses to date anyone who “isn’t Bayesian”. What matters is that Bayes is cool, and if you don’t know Bayes, you aren’t cool.

Why does a mathematical concept generate this strange enthusiasm in its students? What is the so-called Bayesian Revolution now sweeping through the sciences, which claims to subsume even the experimental method itself as a special case? What is the secret that the adherents of Bayes know? What is the light that they have seen?

Soon you will know. Soon you will be one of us.

The essay begins with this example problem:

1% of women at age forty who participate in routine screening have breast cancer. 80% of women with breast cancer will get positive mammographies. 9.6% of women without breast cancer will also get positive mammographies. A woman in this age group had a positive mammography in a routine screening. What is the probability that she actually has breast cancer?

Most doctors estimate between 70% to 80%, which is wildly incorrect. The correct answer is 7.8%. Only about 1/6th of doctors get the right answer.

You’ll get the wrong answer if you don’t know Bayes. So read the essay and learn yourself some Bayes.

And if you made it all the way through that, you’re probably the type that will get value from A Technical Explanation of Bayes’ Theorem.

Previous post:

Next post:

{ 9 comments… read them below or add one }

Eneasz December 8, 2010 at 10:13 am

A few months ago a different poster at Less Wrong put up an Illustrated Explanation of Bayes Theorem.

It covers the same basic information but does so visually rather than via many words, which made it quite a bit easier for me (and I suspect many others). I wouldn’t say it’s better, it’s just different. And understanding Bayes is probably easier if you read both.


Eneasz December 8, 2010 at 10:16 am

From the Illustrated Explanation:
“I like to think in very abstract terms. What this means in practice is that, if there’s some simple, general, elegant point to be made, tell it to me right away. Don’t start with some messy concrete example and attempt to “work upward”, in the hope that difficult-to-grasp abstract concepts will be made more palatable by relating them to “real life”. If you do that, I’m liable to get stuck in the trees and not see the forest.”

(Sorry for two seperate comments on same subject)


Josh December 8, 2010 at 3:21 pm

As a quasi-statistician who frequently uses Bayesian techniques, I find this kind of thing pretty strange. I’ve found Bayesian statistics useful for a number of reasons, but it’s really just a matter of what you need to do. I think a lot of people are overly excited about Bayesianism…


Luke Muehlhauser December 8, 2010 at 5:58 pm


It depends a lot on whether your are an objective Bayesian or a subjective Bayesian, I suppose. I lean toward Objective Bayesianism, currently.


Patrick December 8, 2010 at 6:32 pm

Josh- for me it was realizing that falsifiability wasn’t actually the end-all of scientific inquiry. You could scientifically investigate unfalsifiable scenarios if you started considering them in terms of likelihood. I learned that about the same point that I was figuring out on my own that many deductive fallacies are actually proper induction, so it fit together very neatly.

At this point the fun is in seeing how Bayes, like deduction and falsifiability, can be abused by people who want to sell you things.


Luke Muehlhauser December 8, 2010 at 6:43 pm


Yup. I’m attracted to the whole project of seeing decision theory, testability, Occam’s razor, and explanationism all ‘explained’ (and justified) within the Bayesian framework. The point is this: We’re doing math when we say ‘It’s like that,’ or ‘It’s seems like this fits better with the data than this other theory’, it’s just that when you plug your uncertainty into Bayes then you admit you’re doing math, and can accurately model your probabilities and your error bars.

Hence, formal epistemology.


Josh December 8, 2010 at 7:46 pm

I dunno. For me, there are two main reasons I use Bayes in my research: first, it completely gets rid of multiple testing problems, which are a huge pain and second, (in my particular application) it allows me to avoid a model choice issue because frequentists simply can’t do one of the things that I do (I use what’s called a Dirichlet process mixture to infer the number of components in a mixture model, which frequentists can only do if they use insane model choice things).

However, I still feel like there are so many weird issues with Bayesian statistics that really need to be dealt with. For example, despite claims to the contrary, we still really just go with priors that suit our purposes. Conjugacy is the name of the game in a great many cases (my own included), and most priors are just completely unmotivated (e.g. exponential priors on branch lengths in Bayesian phylogenetics). Things like marginal likelihoods and marginalization over nuisance parameters seem cool, but in practice don’t really do much better than comparable frequentist techniques and can do much much worse if you misspecify a prior (in spite of the Bernstein-Von Mises theorem).


Joseph December 9, 2010 at 3:57 am

Yep, christians love Bayesianism as they can misuse it to prove that miracles are possible. You find such idiocy on the Net along with evolution deniers, ID proponents and great attempts to show that Relativity is wrong. It’s sickening.


Markus Ramikin February 5, 2011 at 9:55 pm

Hey, don’t link to Yudkowski’s article on the Bayes theorem and at the same time spoil the problem that he wants the reader to think about first (the mammography thing, you posted the answer immediately under for no intelligent reason).


Leave a Comment