Reading Yudkowsky, part 50

by Luke Muehlhauser on June 30, 2011 in Eliezer Yudkowsky,Resources,Reviews

AI researcher Eliezer Yudkowsky is something of an expert at human rationality, and at teaching it to others. His hundreds of posts at Less Wrong are a treasure trove for those who want to improve their own rationality. As such, I’m reading all of them, chronologically.

I suspect some of my readers want to “level up” their rationality, too. So I’m keeping a diary of my Yudkowsky reading. Feel free to follow along.

His 419th post is Whither Moral Progress? After that, The Gift We Give Tomorrow examines why evolution would produce morally motivated creatures. Could Anything Be Right? argues that we shouldn’t just be content with ignorance about moral facts. We should try to figure it out, if we can.

Existential Angst Factory offers some words for getting over existential angst. Touching the Old is a quick aside. Next: If physics can blow us up, Should We Ban Physics?

Fake Norms, or “Truth” vs. Truth brings to mind Stephen Colbert’s term, truthiness:

I don’t think there does exist any social norm in favor of truth.  There’s a social norm in favor of “truth”.  There’s a difference.

How would a norm in favor of truth actually be expressed, or acquired?

…If you heard someone say “I don’t care about the evidence, I just want to believe in God”, and you saw everyone else in the room gasp and regard them in frozen shock, then your brain would generalize a social norm against self-deception…

If, on the other hand, you see lots of people saying “Isn’t the truth wonderful?” or “I am in favor of truth”, then you learn that when someone says “truth”, you are supposed to applaud.

Now there are certain particular cases where someone will be castigated if they admit they refuse to see the truth: for example, “I’ve seen the evidence on global warming but I don’t want to believe it.”  You couldn’t get away with that in modern society.  But this indignation doesn’t have to derive from violating a norm in favor of truth – it can derive from the widely held norm, “‘global warming’ is bad”.

Next, Eliezer explains When (Not) To Use Probabilities:

It may come as a surprise to some readers of this blog, that I do not always advocate using probabilities.

Or rather, I don’t always advocate that human beings, trying to solve their problems, should try to make up verbal probabilities, and then apply the laws of probability theory or decision theory to whatever number they just made up, and then use the result as their final belief or decision.

Not sure where a flying ball will land?  I don’t advise trying to formulate a probability distribution over its landing spots, performing deliberate Bayesian updates on your glances at the ball, and calculating the expected utility of all possible strings of motor instructions to your muscles.

Trying to catch a flying ball, you’re probably better off with your brain’s built-in mechanisms, then using deliberative verbal reasoning to invent or manipulate probabilities.

But this doesn’t mean you’re going beyond probability theory or above probability theory.

Then, Eliezer asks: Can Counterfactuals Be True?

The classic explanation of counterfactuals begins with this distinction:

  1. If Lee Harvey Oswald didn’t shoot John F. Kennedy, then someone else did.
  2. If Lee Harvey Oswald hadn’t shot John F. Kennedy, someone else would have.

In ordinary usage we would agree with the first statement, but not the second (I hope).

If, somehow, we learn the definite fact that Oswald did not shoot Kennedy, then someone else must have done so, since Kennedy was in fact shot.

But if we went back in time and removed Oswald, while leaving everything else the same, then – unless you believe there was a conspiracy – there’s no particular reason to believe Kennedy would be shot:

We start by imagining the same historical situation that existed in 1963 – by a further act of imagination, we remove Oswald from our vision – we run forward the laws that we think govern the world – visualize Kennedy parading through in his limousine – and find that, in our imagination, no one shoots Kennedy.

It’s an interesting question whether counterfactuals can be true or false. We never get to experience them directly.

If we disagree on what would have happened if Oswald hadn’t been there, what experiment could we perform to find out which of us is right?

Hint: the answer is “yes.”

Math is Subjunctively Objective reveals (again) that Yudkowsky apparently doesn’t think he has a solution to the problem of universals, even though he thinks he has solutions to the problems of quantum mechanics, rationality, free will, and so much else.

Does Morality Care What You Think? is another meta-ethics dialogue. Changing Your Metaethics tries to open readers to the possibility of changing their metaethics. Next, Eliezer goes about Setting Up Metaethics:

So what we should want, ideally, is a metaethic that:

  1. Adds up to moral normality, including moral errors, moral progress, and things you should do whether you want to or not;
  2. Fits naturally into a non-mysterious universe, postulating no exception to reductionism;
  3. Does not oversimplify humanity’s complicated moral arguments and many terminal values;
  4. Answers all the impossible questions.

Which finally leads to The Meaning of Right, perhaps Eliezer’s central post on morality, but it is long and rambling and does not lead to a short explanation of what Eliezer’s positions are.

Update: I wrote this post months ago. I’ve since gotten clearer on Eliezer’s meta-ethics thanks to this discussion.

Interpersonal Morality concludes:

The question of how much in-principle agreement would exist among human beings about the transpersonal portion of their values, given perfect knowledge of the facts and perhaps a much wider search of the argument space, is not a matter on which we can get much evidence by observing the prevalence of moral agreement and disagreement in today’s world.  Any disagreement might be something that the truth could destroydependent on a different view of how the world is, or maybe just dependent on having not yet heard the right argument.  It is also possible that knowing more could dispel illusions of moral agreement, not just produce new accords.

Humans in Funny Suits remarks that movie aliens always look pretty much like humans, but this is a failure to realize humanity’s own uniqueness. Related: Detached Lever Fallacy.

A Genius for Destruction remarks that highly intelligent people are often just really smart at being stupid – for example, at deceiving themselves, or at showing what biases everyone else is prey to without being able to see their own biases.

The Comedy of Behaviorism is a smack-down of behaviorism. No Logical Positivist I explains why Eliezer is not a logical positivist.

Anthropomorphic Optimism and Contaminated by Optimism warn against a common problem:

The core fallacy of anthropomorphism is expecting something to be predicted by the black box of your brain, when its casual structure is so different from that of a human brain, as to give you no license to expect any such thing.

Hiroshima Day offers some thoughts on mankind’s decision to use atomic fission weapons on human targets.

Previous post:

Next post:

{ 5 comments… read them below or add one }

Kyle Foley June 30, 2011 at 9:06 am

are you going to stand up for yourself and defend your atheism? I’ve just written a devastating critique of your fine-tuning analysis. Are you going to simply shut that out of your mind?


cl June 30, 2011 at 2:08 pm


Do you have a link to your devastating critique? I’d like to check it out.


cl June 30, 2011 at 2:29 pm

A few afterthoughts, in the form of questions:

1) As atheism is a negative position, what is there to defend?

2) Since Luke has declared both the existence of God and philosophy of mind as “settled issues,” should anybody really be surprised that he no longer entertains intelligent discourse on these matters?


Kyle Foley July 1, 2011 at 7:58 am

I sent it through the contact sheet on your website. Thank you for asking me for this in a respectful way. It was wrong of me to use the loaded term devastating and I apologize for it. I look forward to a healthy and polite exchange of ideas between us. If you can’t find the paper in your email, then I’ll post it online.


meh July 1, 2011 at 10:12 am

Yudkowsky = boring shit.


Leave a Comment