AI researcher Eliezer Yudkowsky is something of an expert at human rationality, and at teaching it to others. His hundreds of posts at Less Wrong are a treasure trove for those who want to improve their own rationality. As such, I’m reading all of them, chronologically.
I suspect some of my readers want to “level up” their rationality, too. So I’m keeping a diary of my Yudkowsky reading. Feel free to follow along.
A recent conversation reminded me of this simple, important, and difficult method:
When someone asks you “Why are you doing X?”,
And you don’t remember an answer previously in mind,
Do not ask yourself “Why am I doing X?”.
For example, if someone asks you
“Why are you using a QWERTY keyboard?” or “Why haven’t you invested in stocks?”
and you don’t remember already considering this exact question and deciding it,
do not ask yourself “Why am I using a QWERTY keyboard?” or “Why aren’t I invested in stocks?”
“Should I do X, or not?”
Should I use a QWERTY keyboard, or not? Should I invest in stocks, or not?
When you finish considering this question, print out a traceback of the arguments that you yourself considered in order to arrive at your decision, whether that decision is to X, or not X. Those are your only real reasons, nor is it possible to arrive at a real reason in any other way.
The series on AI that I don’t have time to read continues with Recognizing Intelligence and Lawful Creativity, Lawful Uncertainty, Worse Than Random, The Weighted Majority Problem, Selling Non-Apples. (There are also a few asides tucked in there: 1, 2, 3, 4.)
Of more universal interest is The Nature of Logic, though unfortunately it is written in a non-universal language, that of computer programming. Then, the AI thread continues with Logical or Connectionist AI, Failure by Analogy, and Failure by Affective Analogy.
Next comes a series concerning “The Hanson-Yudkowsky Al-Foom Debate”:
- The Weak Inside View
- The First World Takeover
- Whence Your Abstractions
- Observing Optimization
- Life’s Story Continues
- Surprised by Brains
- Cascades, Cycles, Insight
- Recursion, Magic
Far more accessible than these is The Complete Idiot’s Guide to Ad Hominem, which quotes Stephen Bond:
In reality, ad hominem is unrelated to sarcasm or personal abuse. Argumentum ad hominem is the logical fallacy of attempting to undermine a speaker’s argument by attacking the speaker instead of addressing the argument. The mere presence of a personal attack does not indicate ad hominem: the attack must be used for the purpose of undermining the argument, or otherwise the logical fallacy isn’t there.
A: “All rodents are mammals, but a weasel isn’t a rodent, so it can’t be a mammal.”
B: “You evidently know nothing about logic. This does not logically follow.”
B’s argument is still not ad hominem. B does not imply that A’s sentence does not logically follow because A knows nothing about logic. B is still addressing the substance of A’s argument…
And then, the Singularity and AI stuff that I’m not reading for now, continues:
- Englebart: Insufficiently Recursive
- Total Nano Domination
- Chaotic Inversion
- Singletons Rule OK
- Disappointment in the Future
- Recursive Self-Improvement
- Hard Takeoff
- Permitted Possibility & Locality
- Underconstrained Abstractions
- Sustained Strong Recursion
- Is That Your True Rejection?
- Artificial Mysterious Intelligence
- True Sources of Disagreement
- Disjunctions, Antipredictions, Etc.
Also tucked in there is The Mechanics of Disagreement:
Two ideal Bayesians cannot have common knowledge of disagreement; this is a theorem. If two rationalist-wannabes have common knowledge of a disagreement between them, what could be going wrong?
The obvious interpretation of these theorems is that if you know that a cognitive machine is a rational processor of evidence, its beliefs become evidence themselves.
If you design an AI and the AI says “This fair coin came up heads with 80% probability”, then you know that the AI has accumulated evidence with an likelihood ratio of 4:1 favoring heads – because the AI only emits that statement under those circumstances.
It’s not a matter of charity; it’s just that this is how you think the other cognitive machine works.
And if you tell an ideal rationalist, “I think this fair coin came up heads with 80% probability”, and they reply, “I now think this fair coin came up heads with 25% probability”, and your sources of evidence are independent of each other, then you should accept this verdict, reasoning that (before you spoke) the other mind must have encountered evidence with a likelihood of 1:12 favoring tails.
But this assumes that the other mind also thinks that you’re processing evidence correctly, so that, by the time it says “I now think this fair coin came up heads, p=.25″, it has already taken into account the full impact of all the evidence you know about, before adding more evidence of its own.
If, on the other hand, the other mind doesn’t trust your rationality, then it won’t accept your evidence at face value, and the estimate that it gives won’t integrate the full impact of the evidence you observed.
So does this mean that when two rationalists trust each other’s rationality less than completely, then they can agree to disagree?
It’s not that simple. Rationalists should not trust themselves entirely, either.