AI researcher Eliezer Yudkowsky is something of an expert at human rationality, and at teaching it to others. His hundreds of posts at Less Wrong are a treasure trove for those who want to improve their own rationality. As such, I’m reading all of them, chronologically.
I suspect some of my readers want to “level up” their rationality, too. So I’m keeping a diary of my Yudkowsky reading. Feel free to follow along.
His 400th post is What Would You Do Without Morality?
To those who say “Nothing is real,” I once replied, “That’s great, but how does the nothing work?”
Suppose you learned, suddenly and definitively, that nothing is moral and nothing is right; that everything is permissible and nothing is forbidden.
Devastating news, to be sure – and no, I am not telling you this in real life. But suppose I did tell it to you. Suppose that, whatever you think is the basis of your moral philosophy, I convincingly tore it apart, and moreover showed you that nothing could fill its place. Suppose I proved that all utilities equaled zero.
I know that Your-Moral-Philosophy is as true and undisprovable as 2 + 2 = 4. But still, I ask that you do your best to perform the thought experiment, and concretely envision the possibilities even if they seem painful, or pointless, or logically incapable of any good reply.
Would you still tip cabdrivers? Would you cheat on your Significant Other? If a child lay fainted on the train tracks, would you still drag them off?
Next, The Moral Void:
And if an external objective morality does say that the universe should occupy some horrifying state… let’s not even ask what you’re going to do about that. No, instead I ask: What would you have wished for the external objective morality to be instead? What’s the best news you could have gotten, reading that stone tablet?
Go ahead. Indulge your fantasy. Would you want the stone tablet to say people should die of old age, or that people should live as long as they wanted? If you could write the stone tablet yourself, what would it say?
Maybe you should just do that?
I mean… if an external objective morality tells you to kill people, why should you even listen?
Discussions of morality seem to me to often end up turning around two different intuitions, which I might label morality-as-preference and morality-as-given. The former crowd tends to equate morality with what people want; the latter to regard morality as something you can’t change by changing people.
As for me, I have my own notions, which I am working up to presenting. But above all, I try to avoid avoiding difficult questions. Here are what I see as (some of) the difficult questions for the two intuitions:
- For morality-as-preference:
- Why do people seem to mean different things by “I want the pie” and “It is right that I should get the pie”? Why are the two propositions argued in different ways?
- When and why do people change their terminal values? Do the concepts of “moral error” and “moral progress” have referents? Why would anyone want to change what they want?
- Why and how does anyone ever “do something they know they shouldn’t”, or “want something they know is wrong”? Does the notion of morality-as-preference really add up to moral normality?
- For morality-as-given:
- Would it be possible for everyone in the world to be wrong about morality, and wrong about how to update their beliefs about morality, and wrong about how to choose between metamoralities, etcetera? So that there would be a morality, but it would be entirely outside our frame of reference? What distinguishes this state of affairs, from finding a random stone tablet showing the words “You should commit suicide”?
- How does a world in which a moral proposition is true, differ from a world in which that moral proposition is false? If the answer is “no”, how does anyone perceive moral givens?
- Is it better for people to be happy than sad? If so, why does morality look amazingly likegodshatter of natural selection?
- Am I not allowed to construct an alien mind that evaluates morality differently? What will stop me from doing so?
If every belief must be justified, and those justifications in turn must be justified, then how is the infinite recursion terminated?
And if you’re allowed to end in something assumed-without-justification, then why aren’t you allowed to assumeanything without justification?
A similar critique is sometimes leveled against Bayesianism – that it requires assuming some prior – by people who apparently think that the problem of induction is a particular problem of Bayesianism, which you can avoid by using classical statistics. I will speak of this later, perhaps.
Bayesian updating won’t solve the problem of induction, but…
There are possible minds in mind design space who have anti-Occamian and anti-Laplacian priors; they believe that simpler theories are less likely to be correct, and that the more often something happens, the less likely it is to happen again.
And when you ask these strange beings why they keep using priors that never seem to work in real life… they reply, “Because it’s never worked for us before!”
Now, one lesson you might derive from this, is “Don’t be born with a stupid prior.” This is an amazingly helpful principle on many real-world problems, but I doubt it will satisfy philosophers.
…Should I trust Occam’s Razor? Well, how well does (any particular version of) Occam’s Razor seem to work in practice? What kind of probability-theoretic justifications can I find for it? When I look at the universe, does it seem like the kind of universe in which Occam’s Razor would work well?
Should I trust my brain? Obviously not; it doesn’t always work. But nonetheless, the human brain seems much more powerful than the most sophisticated computer programs I could consider trusting otherwise. How well does my brain work in practice, on which sorts of problems?
When I examine the causal history of my brain – its origins in natural selection – I find, on the one hand, all sorts of specific reasons for doubt; my brain was optimized to run on the ancestral savanna, not to do math. But on the other hand, it’s also clear why, loosely speaking, it’s possible that the brain really could work. Natural selection would have quickly eliminated brains so completely unsuited to reasoning, so anti-helpful, as anti-Occamian or anti-Laplacian priors.
So what I did in practice, does not amount to declaring a sudden halt to questioning and justification. I’m not halting the chain of examination at the point that I encounter Occam’s Razor, or my brain, or some other unquestionable. The chain of examination continues – but it continues, unavoidably, using my current brain and my current grasp on reasoning techniques. What else could I possibly use?
Indeed, no matter what I did with this dilemma, it would be me doing it. Even if I trusted something else, like some computer program, it would be my own decision to trust it.
But this is not a resolution of the problem:
…wouldn’t it be nice if there were some way to justify using Occam’s Razor, or justify predicting that the future will resemble the past, without assuming that those methods of reasoning which have worked on previous occasions are better than those which have continually failed?
Wouldn’t it be nice if there were some chain of justifications that neither ended in an unexaminable assumption, nor was forced to examine itself under its own rules, but, instead, could be explained starting from absolute scratch to an ideal philosophy student of perfect emptiness?
Well, I’d certainly be interested, but I don’t expect to see it done any time soon.
So, at the end of the day, what happens when someone keeps asking me “Why do you believe what you believe?”
At present, I start going around in a loop at the point where I explain, “I predict the future as though it will resemble the past on the simplest and most stable level of organization I can identify, because previously, this rule has usually worked to generate good results; and using the simple assumption of a simple universe, I can see why it generates good results; and I can even see how my brain might have evolved to be able to observe the universe with some degree of accuracy, if my observations are correct.”
But then… haven’t I just licensed circular logic?
Actually, I’ve just licensed reflecting on your mind’s degree of trustworthiness, using your current mind as opposed to something else.
…Everything, without exception, needs justification. Sometimes – unavoidably, as far as I can tell – those justifications will go around in reflective loops. I do think that reflective loops have a meta-character which should enable one to distinguish them, by common sense, from circular logics. But anyone seriously considering a circular logic in the first place, is probably out to lunch in matters of rationality; and will simply insist that their circular logic is a “reflective loop” even if it consists of a single scrap of paper saying “Trust me”. Well, you can’t always optimize your rationality techniques according to the sole consideration of preventing those bent on self-destruction from abusing them.
The important thing is to hold nothing back in your criticisms of how to criticize; nor should you regard the unavoidability of loopy justifications as a warrant of immunity from questioning.
Always apply full force, whether it loops or not – do the best you can possibly do, whether it loops or not – and play, ultimately, to win.