Reading Yudkowsky, part 5

by Luke Muehlhauser on December 6, 2010 in Eliezer Yudkowsky,Resources,Reviews

AI researcher Eliezer Yudkowsky is something of an expert at human rationality, and at teaching it to others. His hundreds of posts at Overcoming Bias (now moved to Less Wrong) are a treasure trove for those who want to improve their own rationality. As such, I’m reading all of them, chronologically.

I suspect some of my readers want to improve their rationality, too. So I’m keeping a diary of my Yudkowsky reading. Feel free to follow along.

Yudkowsky’s 30th post is Inductive Bias, where he points out that whereas statistical bias is always bad (though sometimes useful), inductive bias can sometimes be good (if the bias is true).

The next post is just a call for Suggested Posts. More interesting is Futuristic Predictions as Consumable Goods. Yudkowsky wonders about people like Tom Friedman who, every six months, says that “the next six months” are the most crucial months of the Iraq war. Or what about AI researchers who say “the next 3-5 years” are the most important?

Why do futurists make the same mistaken predictions over and over? The same reason politicians abandon campaign promises and switch principles as expediency demands. Predictions, like promises, are sold today and consumed today. They produce a few chewy bites of delicious optimism or delicious horror, and then they’re gone. If the tastiest prediction is allegedly about a time interval “3-5 years in the future” (for AI projects) or “6 months in the future” (for Iraq), then futurists will produce tasty predictions of that kind. They have no reason to change the formulation any more than Hershey has to change the composition of its chocolate bars. People won’t remember the prediction in 6 months or 3-5 years, any more than chocolate sits around in your stomach for a year and keeps you full.

Marginally Zero-Sum Efforts applies the problem of the commons to the writing of grant proposals. Lotteries: A Waste of Hope supposes that lotteries are not just a money sink, but

a sink of emotional energy. It encourages people to invest their dreams, their hopes for a better future, into an infinitesimal probability. If not for the lottery, maybe they would fantasize about going to technical school, or opening their own business, or getting a promotion at work – things they might be able to actually do, hopes that would make them want to become stronger.

Which leads to the more general note:

The process of overcoming bias requires (1) first noticing the bias, (2) analyzing the bias in detail, (3) deciding that the bias is bad, (4) figuring out a workaround, and then (5) implementing it. It’s unfortunate how many people get through steps 1 and 2 and then bog down in step 3, which by rights should be the easiest of the five. Biases are lemons, not lemonade, and we shouldn’t try to make lemonade out of them – just burn those lemons down.

In Priors as Mathematical Objects, Yudkowsky proposes that we could see inductive learning as a mathematical object – say, as a Python program.

Some people defend lotteries by saying people are not so much buying a chance at winning but a fantasy that has value to them. But if that’s the case, we could design a New Improved Lottery that would facilitate fantasizing better.

And really, if people want to buy lottery tickets, who am I to criticize their decision? In Your Rationality is My Business, Yudkowsky writes:

What business is it of mine, if someone else chooses to believe what is pleasant rather than what is true? Can’t we each choose for ourselves whether to care about the truth?

…One of [my] interests is the human pursuit of truth… And that makes your rationality my business.

Is this a dangerous idea? Yes, and not just pleasantly edgy “dangerous”. People have been burned to death because some priest decided that they didn’t think the way they should…

Here’s my proposal: Let’s argue against bad ideas but not set their bearers on fire…

I cannot help but care how you think, because – as I cannot help but see the universe – each time a human being turns away from the truth, the unfolding story of humankind becomes a little darker…

The Consolidated Nature of Morality Thread lists 10 major debates/conundrums in moral theory:

  1. It certainly looks like there is an important distinction between a statement like “The total loss of human life caused by World War II was roughly 72 million people” and “We ought to avoid a repeat of World War II.” Anyone who argues that these statements are of the same fundamental kind must explain away the apparent structural differences between them. What are the exact structural differences?
  2. We experience some of our morals and preferences as being voluntary choices, others as involuntary perceptions. I choose to play on the side of Rationality, but I don’t think I could choose to believe that death is good any more than I could choose to believe the sky is green. What psychological factors account for these differences in my perceptions of my own preferences?
  3. At a relatively young age, children begin to believe that while the teacher can make it all right to stand on your chair by giving permission, the teacher cannot make it all right to steal from someone else’s backpack. (I can’t recall the exact citation on this.) Do young children in a religious environment believe that God can make it all right to steal from someone’s backpack?
  4. Both individual human beings and civilizations appear to change at least some of their moral beliefs over the course of time. Some of these changes are experienced as “decisions”, others are experienced as “discoveries”. Is there a systematic direction to at least some of these changes? How does this systematic direction arise causally?
  5. To paraphrase Alfred Tarski, the statement “My car is painted green” is true if and only if my car is painted green. Similarly, someone might try to get away with asserting that the statement “Human deaths are bad” is true if and only if human deaths are bad. Is this valid?
  6. Suppose I involuntarily administered to you a potion which would cause you to believe that human deaths were good. Afterward, would you believe truly that human deaths were good, or would you believe falsely that human deaths were good?
  7. Although the statement “My car is painted green” is presently false, I can make it true at a future time by painting my car green. However, I can think of no analogous action I could take which would make it right to kill people. Does this make the moral statement stronger, weaker, or is there no sense in making the comparison?
  8. There does not appear to be any “place” in the environment where the referents of moral statements are stored, analogous to the place where my car is stored. Does this necessarily indicate that moral statements are empty of content, or could they correspond to something else? Is the statement 2 + 2 = 4 true? Could it be made untrue? Is it falsifiable? Where is its content?
  9. The phrase “is/ought” gap refers to the notion that no ought statement can be logically derived from any number of is statements, without at least one ought statement in the mix. For example, suppose I have a remote control with two buttons, and the red button kills an innocent prisoner, and the green button sets them free. I cannot derive the ought-statement, “I ought not to press the red button”, without both the is-statement “If I press the red button, an innocent will die” and the ought-statement “I ought not to kill innocents.” Should we distinguish mixed ought-statements like “I ought not to press the red button” from pure ought-statements like “I ought not to kill innocents”? If so, is there really any such thing as a “pure” ought-statement, or do they all have is-statements mixed into them somewhere?
  10. The statement “This painting is beautiful” could be rendered untrue by flinging a bucket of mud on the painting. Similarly, in the remote-control example above, the statement “It is wrong to press the red button” can be rendered untrue by rewiring the remote. Are there pure aesthetic judgments? Are there pure preferences?

I am tempted to make my own list that captures the debates even better, but this is a decent place to start.

Previous post:

Next post:

{ 5 comments… read them below or add one }

Joseph December 6, 2010 at 3:51 pm

In Number 5, “My car is painted green” is true if and only if my car is painted green. Similarly, someone might try to get away with asserting that the statement “Human deaths are bad” is true if and only if human deaths are bad. Is this valid?

This is like mixing apples with oranges. The statement, “My car is painted green”, can be verified empirically. OTOH, the statement, “Human deaths are bad”, cannot. To an alien from outer space, it could disagree with the 2nd statement and human deaths might be good, or if the human population on planet earth would reach ONE TRILLION, perhaps many among us would consider that human deaths might be good.


Luke Muehlhauser December 6, 2010 at 8:16 pm


Well, that depends on whether or not you are using the word-tool of “bad” to refer to a natural property or not…


Taranu December 7, 2010 at 12:59 am

Please do make your “own list that captures the debates even better”


Dabs December 7, 2010 at 3:58 am

This new theme is way past unreadable. Terrible…


Curt March 19, 2011 at 1:41 pm

I think that E.Y’s comments on April 15th 2007 were just plain brilliant.
Just Plain Curt


Leave a Comment