Reading Yudkowsky, part 48

by Luke Muehlhauser on June 23, 2011 in Eliezer Yudkowsky,Resources,Reviews

AI researcher Eliezer Yudkowsky is something of an expert at human rationality, and at teaching it to others. His hundreds of posts at Less Wrong are a treasure trove for those who want to improve their own rationality. As such, I’m reading all of them, chronologically.

I suspect some of my readers want to “level up” their rationality, too. So I’m keeping a diary of my Yudkowsky reading. Feel free to follow along.

His 400th post is What Would You Do Without Morality?

To those who say “Nothing is real,” I once replied, “That’s great, but how does the nothing work?”

Suppose you learned, suddenly and definitively, that nothing is moral and nothing is right; that everything is permissible and nothing is forbidden.

Devastating news, to be sure – and no, I am not telling you this in real life.  But suppose I did tell it to you.  Suppose that, whatever you think is the basis of your moral philosophy, I convincingly tore it apart, and moreover showed you that nothing could fill its place.  Suppose I proved that all utilities equaled zero.

I know that Your-Moral-Philosophy is as true and undisprovable as 2 + 2 = 4. But still, I ask that you do your best to perform the thought experiment, and concretely envision the possibilities even if they seem painful, or pointless, or logically incapable of any good reply.

Would you still tip cabdrivers?  Would you cheat on your Significant Other?  If a child lay fainted on the train tracks, would you still drag them off?

Next, The Moral Void:

And if an external objective morality does say that the universe should occupy some horrifying state… let’s not even ask what you’re going to do about that.  No, instead I ask:  What would you have wished for the external objective morality to be instead?  What’s the best news you could have gotten, reading that stone tablet?

Go ahead.  Indulge your fantasy.  Would you want the stone tablet to say people should die of old age, or that people should live as long as they wanted?  If you could write the stone tablet yourself, what would it say?

Maybe you should just do that?

I mean… if an external objective morality tells you to kill people, why should you even listen?

Created Already in Motion considers Lewis Carroll’s classic story What the Tortoise Said to Achilles.

I’d Take It is a quick aside, and then The Bedrock of Fairness is a dialogue about how to define justice and morality.

After another quick aside, and then Moral Complexities presents some of the questions moral theorists must answer:

Discussions of morality seem to me to often end up turning around two different intuitions, which I might label morality-as-preference and morality-as-given.  The former crowd tends to equate morality with what people want; the latter to regard morality as something you can’t change by changing people.

As for me, I have my own notions, which I am working up to presenting.  But above all, I try to avoid avoiding difficult questions.  Here are what I see as (some of) the difficult questions for the two intuitions:

  • For morality-as-preference:
    • Why do people seem to mean different things by “I want the pie” and “It is right that I should get the pie”?  Why are the two propositions argued in different ways?
    • When and why do people change their terminal values?  Do the concepts of “moral error” and “moral progress” have referents?  Why would anyone want to change what they want?
    • Why and how does anyone ever “do something they know they shouldn’t”, or “want something they know is wrong”?  Does the notion of morality-as-preference really add up to moral normality?
  • For morality-as-given:
    • Would it be possible for everyone in the world to be wrong about morality, and wrong about how to update their beliefs about morality, and wrong about how to choose between metamoralities, etcetera?  So that there would be a morality, but it would be entirely outside our frame of reference?  What distinguishes this state of affairs, from finding a random stone tablet showing the words “You should commit suicide”?
    • How does a world in which a moral proposition is true, differ from a world in which that moral proposition is false?  If the answer is “no”, how does anyone perceive moral givens?
    • Is it better for people to be happy than sad?  If so, why does morality look amazingly likegodshatter of natural selection?
    • Am I not allowed to construct an alien mind that evaluates morality differently?  What will stop me from doing so?

Then, two more dialogues: Is Morality Preference? and Is Morality Given?

Will As Thou Wilt is another quick aside, and then Where Recursive Justification Hits Bottom tackles the problem of induction:

If every belief must be justified, and those justifications in turn must be justified, then how is the infinite recursion terminated?

And if you’re allowed to end in something assumed-without-justification, then why aren’t you allowed to assumeanything without justification?

A similar critique is sometimes leveled against Bayesianism – that it requires assuming some prior – by people who apparently think that the problem of induction is a particular problem of Bayesianism, which you can avoid by using classical statistics.  I will speak of this later, perhaps.

Bayesian updating won’t solve the problem of induction, but…

There are possible minds in mind design space who have anti-Occamian and anti-Laplacian priors; they believe that simpler theories are less likely to be correct, and that the more often something happens, the less likely it is to happen again.

And when you ask these strange beings why they keep using priors that never seem to work in real life… they reply, “Because it’s never worked for us before!”

Now, one lesson you might derive from this, is “Don’t be born with a stupid prior.”  This is an amazingly helpful principle on many real-world problems, but I doubt it will satisfy philosophers.

Should I trust Occam’s Razor?  Well, how well does (any particular version of) Occam’s Razor seem to work in practice?  What kind of probability-theoretic justifications can I find for it?  When I look at the universe, does it seem like the kind of universe in which Occam’s Razor would work well?

Should I trust my brain?  Obviously not; it doesn’t always work.  But nonetheless, the human brain seems much more powerful than the most sophisticated computer programs I could consider trusting otherwise.  How well does my brain work in practice, on which sorts of problems?

When I examine the causal history of my brain – its origins in natural selection – I find, on the one hand, all sorts of specific reasons for doubt; my brain was optimized to run on the ancestral savanna, not to do math.  But on the other hand, it’s also clear why, loosely speaking, it’s possible that the brain really could work.  Natural selection would have quickly eliminated brains so completely unsuited to reasoning, so anti-helpful, as anti-Occamian or anti-Laplacian priors.

So what I did in practice, does not amount to declaring a sudden halt to questioning and justification.  I’m not halting the chain of examination at the point that I encounter Occam’s Razor, or my brain, or some other unquestionable.  The chain of examination continues – but it continues, unavoidably, using my current brain and my current grasp on reasoning techniques.  What else could I possibly use?

Indeed, no matter what I did with this dilemma, it would be me doing it.  Even if I trusted something else, like some computer program, it would be my own decision to trust it.

But this is not a resolution of the problem:

…wouldn’t it be nice if there were some way to justify using Occam’s Razor, or justify predicting that the future will resemble the past, without assuming that those methods of reasoning which have worked on previous occasions are better than those which have continually failed?

Wouldn’t it be nice if there were some chain of justifications that neither ended in an unexaminable assumption, nor was forced to examine itself under its own rules, but, instead, could be explained starting from absolute scratch to an ideal philosophy student of perfect emptiness?

Well, I’d certainly be interested, but I don’t expect to see it done any time soon.

Eliezer concludes:

So, at the end of the day, what happens when someone keeps asking me “Why do you believe what you believe?”

At present, I start going around in a loop at the point where I explain, “I predict the future as though it will resemble the past on the simplest and most stable level of organization I can identify, because previously, this rule has usually worked to generate good results; and using the simple assumption of a simple universe, I can see why it generates good results; and I can even see how my brain might have evolved to be able to observe the universe with some degree of accuracy, if my observations are correct.”

But then… haven’t I just licensed circular logic?

Actually, I’ve just licensed reflecting on your mind’s degree of trustworthiness, using your current mind as opposed to something else.

Everything, without exception, needs justification.  Sometimes – unavoidably, as far as I can tell – those justifications will go around in reflective loops.  I do think that reflective loops have a meta-character which should enable one to distinguish them, by common sense, from circular logics.  But anyone seriously considering a circular logic in the first place, is probably out to lunch in matters of rationality; and will simply insist that their circular logic is a “reflective loop” even if it consists of a single scrap of paper saying “Trust me”.  Well, you can’t always optimize your rationality techniques according to the sole consideration of preventing those bent on self-destruction from abusing them.

The important thing is to hold nothing back in your criticisms of how to criticize; nor should you regard the unavoidability of loopy justifications as a warrant of immunity from questioning.

Always apply full force, whether it loops or not – do the best you can possibly do, whether it loops or not – and play, ultimately, to win.

Previous post:

Next post:

{ 30 comments… read them below or add one }

Alex June 23, 2011 at 4:46 am

Hmm, these posts are becoming a bit pointless. Why just quoting what he says? Where’s the commentary? Where’s the pondering? What’s the point here, apart from just duplicating the content? The old lukeprog wouldn’t stand for this … :)

  (Quote)

Andres June 23, 2011 at 6:12 am

:(

This blog is dead.

Why not just make a final post saying “hey guys, go read all his stuff, this site is now shut down” instead of dragging it on?

  (Quote)

Zeb June 23, 2011 at 6:52 am

Suppose you learned, suddenly and definitively, that nothing is moral and nothing is right; that everything is permissible and nothing is forbidden….
…Suppose I proved that all utilities equaled zero.Would you still tip cabdrivers?…
…Would you cheat on your Significant Other? If a child lay fainted on the train tracks, would you still drag them off?

Hasn’t Grand Theft Auto, or Super Mario World for that matter, answered this for many of us? If it were really the case that morality was false and utility was zero and I knew it beyond a doubt, I expect that I would either commit suicide or just go psychotic. Try to blow everything up or ride a turtle shell until I die, or whatever. The problem is that if moral facts are true, people who ‘believe’ they are not will continue to encounter them and be influenced by them anyway.

  (Quote)

Martin Freedman June 23, 2011 at 8:29 am

I very sadly have to agree this blog is dead. These posts are pointless, if one wants to read EY one can go his posts directly.

Luke, if you have nothing else to write about then still don’t bother with these posts. You have two guest posters, your weekly news bits posts and you can always post here what you post on LW – and you might get different feedback here.

  (Quote)

Zeb June 23, 2011 at 1:17 pm

I like these posts. Considering all there is out there to read and how limited time is, I may never get around to reading EY’s blog “cover to cover.” I’m glad to get a snapshot of what he thinks, even if I know that I don’t really understand it based on these little glimpses. And they do provide jumping off points for interesting conversations in the comments. Since Luke has said that he already wrote all of these and just has them set to post, I think he should let it go on.

  (Quote)

soupsayer June 23, 2011 at 2:42 pm

I don’t mind these posts either. Mostly because I find Yudkowsky’s writing full of arrogance, hubris and to be somewhat “nutcasey”. So, a summary that has weeded some of that stuff out is always welcome.

The non-Yudkowskian part of Luke’s brain did tweet up some links to interesting arguments against the Robo-pocalypse, here and here. The comments are good too. I’m no longer persuaded by Chalmers, although I think his development of personal identity is interesting.

  (Quote)

Robert June 23, 2011 at 9:31 pm

Is Morality Preference? links to Moral Complexities.   Here is the right link.

  (Quote)

Luke Muehlhauser June 24, 2011 at 3:10 am

Robert,

Fixed! Thanks.

  (Quote)

Harold J. June 24, 2011 at 6:54 am

I don’t mind these posts either. Mostly because I find Yudkowsky’s writing full of arrogance, hubris and to be somewhat “nutcasey”. So, a summary that has weeded some of that stuff out is always welcome.

Yeah. I never quite got what Yudkowsky was trying to prove by constantly reminding readers that he read Feynman at age 9.

Also, I wonder if the way the comments are set up at LW reinforce Yudkowsky’s status as something of a cult-leader. For instance, anything that seems slightly critical of Yudkowsky the person, rather than his arguments, is swarmed by his minions and down voted. Now, this is not to suggest ad hominems hurled at Yudkowsky. But if you can’t even question when a leader of a community is being arrogant or a bit delusional then you have a problem. If anything it will probably contribute to a wider divide between commenters – the worshippers of Eliezer and the dissenters. I imagine it will be a difficult gap to overcome given that the dissenters will probably feel justifiable resentment toward being shunned for pointing out a few obvious quirks in Yudkowsky.

  (Quote)

James June 24, 2011 at 11:49 am

Count me in as enjoying these posts. I don’t have time to read all the LW stuff and appreciate the reader’s digest version. Complaining about them seems like complaining about the dinner someone prepared for you — for free by the way.

  (Quote)

Fantomex June 24, 2011 at 2:04 pm

I don’t mind these posts either. Mostly because I find Yudkowsky’s writing full of arrogance, hubris and to be somewhat “nutcasey”. So, a summary that has weeded some of that stuff out is always welcome.

It’s not just arrogance, it’s immaturity. I was able to get past his terribly unfocused writing style, because some of the topics are interesting, but when I got to the post where Yudkowsky tells the story of asking someone if they thought he was the smartest person they’d ever met, I lost all respect for him.

  (Quote)

Robert June 24, 2011 at 3:50 pm

Fantomex, Which post is that?

  (Quote)

Fantomex June 24, 2011 at 5:12 pm

@Robert: It’s “The Level Above Mine”.

  (Quote)

Robert June 24, 2011 at 7:24 pm

Thanks, I just read the post.

It did not strike me the same way. It’s clear he is concerned about his intelligence, but meeting goals to build Friendly AI need some serious intelligence. I look at myself, my own flaws, and feel compelled to cut the guy some slack. Is there some reason you can’t do the same?

  (Quote)

Fantomex June 24, 2011 at 8:34 pm

Taking the post out of the context of the other things I have read of his which lead me to interpret the post as I have, I still don’t understand how it can be read otherwise.

He asked if he was the smartest person that someone had ever met. That question doesn’t seem at all focused on determing if he has sufficient intelligence to solve some problem.

As for “cutting him some slack”: I criticized him in one comment, I don’t believe that to be unreasonable. Your flaws or mine do nothing to change that.

  (Quote)

SuperSamus June 24, 2011 at 11:45 pm

Hey everyone- check out Ed Fesers blog! I pray you will find the one true god!

  (Quote)

Michael June 25, 2011 at 2:52 am

Hey everyone- check out Ed Fesers blog! I pray you will find the one true god!

Lol.

It did not strike me the same way. It’s clear he is concerned about his intelligence, but meeting goals to build Friendly AI need some serious intelligence. I look at myself, my own flaws, and feel compelled to cut the guy some slack. Is there some reason you can’t do the same?

It’s not just arrogance, it’s immaturity.I was able to get past his terribly unfocused writing style, because some of the topics are interesting, but when I got to the post where Yudkowsky tells the story of asking someone if they thought he was the smartest person they’d ever met, I lost all respect for him.

Oh my word, I just read the post. That is just embarrassing! Seriously…I didn’t have a huge amount of respect for Yudkowsky anyway but he is SO up himself! Someone needs a reality check…so Luke why don’t you write a post on the subject of this piece of Yudkowsky’s writing then?? ;P http://lesswrong.com/lw/ua/the_level_above_mine/

  (Quote)

Anonymous June 26, 2011 at 12:14 am

It’s kind of clear that some people here are threatened by productive cyber-contributers and use that tired ol’ troll rhetoric to defame these good people with opinionative crap: “He’s full of himself – he must be – he’s a blogger; his personality sucks; he’s growing a cult; he downvotes my trollish criticisms; blah blah blah boo hoo sob.” Have a tissue.

TO NON-CONTRIBUTING PARASITES AND OTHER TROLLS:
Get hobbies.

TO THE AUTHOR OF THIS BLOG:
Prune your comment box or watch your comments dwindle in quality and sanity.

  (Quote)

Fantomex June 26, 2011 at 12:30 pm

It’s kind of clear that some people here are threatened by productive cyber-contributers and use that tired ol’ troll rhetoric to defame these good people with opinionative crap: “He’s full of himself – he must be – he’s a blogger; his personality sucks; he’s growing a cult; he downvotes my trollish criticisms; blah blah blah boo hoo sob.” Have a tissue.

TO NON-CONTRIBUTING PARASITES AND OTHER TROLLS:
Get hobbies.

TO THE AUTHOR OF THIS BLOG:
Prune your comment box or watch your comments dwindle in quality and sanity.

“cyber-contributers” [sic]? Do you think that just adding a prefix to a word makes it sound better? Contributing to what exactly?

I’m not threatened by whatever a “cyber-contributer” [sic] is, and I’m not defaming anybody. I said I lost respect for him, and gave my reasons. What you’re describing doesn’t seem to accurately represent anyone on this page’s problems with Yudkowsky, so your comment just comes off as mindless fanboyism.

Maybe next time you should calm down a little before posting, and come off a bit more rational.

  (Quote)

antiplastic June 26, 2011 at 2:42 pm

Thank you, Anonymous Cyber-Contributor, for driving the conversation forward in a productive way with your measured and diplomatic tone.

  (Quote)

Dog Diddy June 26, 2011 at 8:51 pm

TO THE AUTHOR OF THIS BLOG:
Prune your comment box or watch your comments dwindle in quality and sanity.

I think your comment is an example of which you speak of. ;)

  (Quote)

hf June 27, 2011 at 12:56 am

After Michael’s reaction I actually went and looked at the post again to see if it said what I remembered. Now I may approach this differently because Eliezer seems like a smarter version of me in particular, rather than a smarter Augustine or Bakunin or what have you. Our similarities allow me to see our difference in intelligence more clearly. So to me the beginning of the post, where he describes the even greater intelligence of E.T. Jaynes, stood out more than the line which Fantomex tripped on.

As for that later part, Eliezer says he asked someone he’d known for a while (not someone he’d just met who he expected to know his work) if this person remembered anyone who seemed smarter than him. Being human, Eliezer presumably wanted to hear an answer of ‘No.’ But failing that, he says, he wanted to hear about someone he might enlist to help solve the problem of an existential risk for humanity. In fact, he explicitly says in that same post that he started Less Wrong in part because some reader might have a better native ability to deal with this problem. That does not sound to me like arrogance. Back to the conversation with Marcello that he relates, I get the sense that (as someone working on the theory of Artificial Intelligence) Eliezer also kept his ears open for some knowledge that might help him understand intelligence better. He might even have wanted some pointers for himself, as the post in question follows one on the topic of Random Anime Motivation #1 ‘I Want To Become Stronger’.

  (Quote)

TK June 28, 2011 at 2:49 pm

all this talk is easily settled by a glance at yudkowsky’s okcupid page. he is, indeed, full of himself.

http://www.okcupid.com/profile/EYudkowsky

  (Quote)

cl June 28, 2011 at 4:28 pm

Hehe… I don’t know why I thought of JP from Grandma’s Boy after reading all this. The salient part comes at 40 sec. I hope somebody else gets a laugh out of it!

  (Quote)

cl June 28, 2011 at 4:33 pm

Although I’m also cool with trophy collection, if you only want to sleep with me once so you can tell your grandchildren about it. -Yudkowsky

LOL!

  (Quote)

Fantomex June 28, 2011 at 9:05 pm

all this talk is easily settled by a glance at yudkowsky’s okcupid page. he is, indeed, full of himself.

I’m not a fan of his either, but I feel like the more fair reaction to his page is “Here is a catastrophically unfunny person. Talking to him must be agonizing” rather than “he thinks he’s so great”.

Although, of course, he does think he’s so great, being a “hero” and all. ;)

  (Quote)

hf June 28, 2011 at 10:46 pm

I’m not a fan of his either, but I feel like the more fair reaction to his page is “Here is a catastrophically unfunny person.

Ha, that just puts you farther from the page’s target audience than me.

(I started out suspecting a fake, by the way, but the humor at least seems like evidence of EY’s authorship. Sort of like how you know Aleister Crowley wrote an essay he attributes to someone else, because if you read it closely the author calls himself a lying snake.)

  (Quote)

Fantomex June 29, 2011 at 1:12 pm

Ha, that just puts you farther from the page’s target audience than me.

I don’t understand.

  (Quote)

cl June 30, 2011 at 1:40 pm

One can be morbidly self-absorbed without being arrogant, but I don’t think one can be arrogant without being morbidly self-absorbed.

  (Quote)

mAST June 30, 2011 at 5:03 pm

all this talk is easily settled by a glance at yudkowsky’s okcupid page. he is, indeed, full of himself.http://www.okcupid.com/profile/EYudkowsky

LOL. I bet he feels depressed that women want him only for his body and not for his mind. What a lonely life. :(

  (Quote)

Leave a Comment