AI researcher Eliezer Yudkowsky is something of an expert at human rationality, and at teaching it to others. His hundreds of posts at Less Wrong are a treasure trove for those who want to improve their own rationality. As such, I’m reading all of them, chronologically.
I suspect some of my readers want to “level up” their rationality, too. So I’m keeping a diary of my Yudkowsky reading. Feel free to follow along.
His 411th post is The Fear of Common Knowledge, about lying and knowing that another knows. My Kind of Reflection attempts to sum up Eliezer’s philosophical method, which is massively influenced by his work in AI.
Next is a post on The Genetic Fallacy:
This is, at first sight, a very strange idea – if the causes of a belief do not determine its systematic reliability, what does? If Deep Blue advises us of a chess move, we trust it based on our understanding of the code that searches the game tree, being unable to evaluate the actual game tree ourselves. What could license any probability assignment as “rational”, except that it was produced by some systematically reliable process?
…The genetic fallacy is formally a fallacy, because the original cause of a belief is not the same as its current justificational status, the sum of all the support and antisupport currently known.
Fundamental Doubts is a nice examination of the phenomenon of doubting. Next is Rebelling Within Nature:
you can’t fight Nature from beyond Nature, only from within it. There is no acausal fulcrum on which to stand outside reality and move it. There is no ghost of perfect emptiness by which you can judge your brain from outside your brain. You can fight the cosmic process, but only by recruiting other abilities that evolution originally gave to you.
And if you fight one emotion within yourself – looking upon your own nature, and judging yourself less than you think should be – saying perhaps, “I should not want to kill my enemies” – then you make that judgment, by…
From within it, naturally.
Eliezer tells of the story when he discovered his parents’ guidebook to parenting:
It described the horrible confusion of the teenage years – teenagers experimenting with alcohol, with drugs, with unsafe sex, with reckless driving, the hormones taking over their minds, the overwhelming importance of peer pressure, the tearful accusations of “You don’t love me!” and “I hate you!”
I took one look at that description, at the tender age of nine, and said to myself in quiet revulsion, I’m not going to do that.
And I didn’t.
My teenage years were not untroubled. But I didn’t do any of the things that the Guide to Parents warned me against. I didn’t drink, drive, drug, lose control to hormones, pay any attention to peer pressure, or ever once think that my parents didn’t love me.
In a safer world, I would have wished for my parents to have hidden that book better.
But in this world, which needs me as I am, I don’t regret finding it.
Probability is Subjectively Objective addresses a common dispute among Bayesians:
E. T. Jaynes, master of the art, once described himself as a subjective-objective Bayesian. Jaynes certainly believed very firmly that probability was in the mind; Jaynes was the one who coined the termMind Projection Fallacy. But Jaynes also didn’t think that this implied a license to make up whatever priors you liked. There was only one correct prior distribution to use, given your state of partial information at the start of the problem.
How can something be in the mind, yet still be objective?
After a long discussion of how minds and calculators compute, Eliezer returns to the question:
Is probability, then, subjective or objective?
Well, probability is computed within human brains or other calculators. A probability is a state of partial information that is possessed by you; if you flip a coin and press it to your arm, the coin is showing heads or tails, but you assign the probability 1/2 until you reveal it. A friend, who got a tiny but not fully informative peek, might assign a probability of 0.6.
…When you think about the ontological nature of probability, and perform reductionism on it – when you try to explain how “probability” fits into a universe in which states of mind do not exist fundamentally – then you find that probability is computed within a brain; and you find that other possible minds could perform mostly-analogous operations with different priors and arrive at different answers.
But, when you consider probability as probability, think about the referent instead of the thought process – which thinking you will do in your own thoughts, which are physical processes – then you will conclude that the vast majority of possible priors are probably wrong. (You will also be able to conceive of priors which are, in fact, better than yours, because they assign more probability to the actual outcome; you just won’t know in advance which alternative prior is the truly better one.)
If you again swap your goggles to think about how probability is implemented in the brain, the seeming objectivity of probability is the way the probability algorithm feels from inside; so it’s no mystery that, considering probability as probability, you feel that it’s not subject to your whims. That’s just what the probability-computation would be expected to say, since the computation doesn’t represent any dependency on your whims.
But when you swap out those goggles and go back to thinking about probabilities, then, by golly, your algorithm seems to be right in computing that probability is not subject to your whims. You can’t win the lottery just by changing your beliefs about it. And if that is the way you would be expected to feel, then so what? The feeling has been explained, not explained away; it is not a mere feeling. Just because a calculation is implemented in your brain, doesn’t mean it’s wrong, after all.
Your “probability that the ten trillionth decimal digit of pi is 4″, is an attribute of yourself, and exists in your mind; the real digit is either 4 or not. And if you could change your belief about the probability by editing your brain, you wouldn’t expect that to change the probability.
Therefore I say of probability that it is “subjectively objective”.