News Bits

by Luke Muehlhauser on January 21, 2012 in News

New at Less Wrong: Can the Chain Still Hold You?

New at Facing the Intelligence Explosion: Value is Complex and Fragile.

Also new links at my Twitter page.

Previous post:

Next post:

{ 9 comments… read them below or add one }

Jonathan Livengood January 21, 2012 at 8:44 am

Your paper “The Intelligence Explosion and Machine Ethics,” linked in “Value is Complex and Fragile,” is well-argued and interesting. Is it under review anywhere or do you have plans to submit it to a journal?

Hopefully constructive criticism to follow:

1. I worry that the distinction between what we want (do) and what we ought to want (do) is not drawn brightly enough in “Value is Complex.” For example, you expect that your reader will react negatively to the death of the firemen. But this reader did not. In that situation, I would rather have one live mother and twelve dead firemen than twelve live firemen and one dead mother. That is a fact about my utilities as I understand them. Maybe I shouldn’t have those utilities, but if so, that is a fact about what I ought to want, not a fact about what I actually do want.

Looking back, a similar thing happens a little bit in the other paper (“The Intelligence Explosion and Machine Ethics”) a little bit. For example, you write: “Our intuitive negative reaction to hypothetical worlds in which pleasure is (more or less) maximized suggests that pleasure is not the only thing we value.” Okay. But if I agree with that, then what? Maybe we ought to value pleasure alone and are making an error here.

2. You give the impression (which I expect you did not intend to give) in both “Value is Complex” and also in “The Singularity” that for most (all?) decision problems, there is an outcome that all humans would prefer. Otherwise, I’m not sure how to make sense of things like this claim from your paper on machine ethics: “we haven’t yet identified a moral theory that, if implemented throughout the universe, would produce a universe we want.” Or like this claim from your chapter on value: “This is also why moral philosophers have spent thousands of years failing to find a simple set of principles that, if enacted, would create a world we want.” The impression one gets is that you think there is some set of people — whoever “we” are — that have a uniquely preferred outcome on universe-optimization tasks. I think this is grist for your mill, insofar as it agrees with the claim that “human values are complex and difficult to specify.” But I think some greater care is called for.

3. I am very skeptical of the claim that there are no safe wishes smaller than an entire human value system. Let me give an example that supposes your Golem Genie. Suppose I say to the Golem, “You are allowed to do only the following two things. From now until April 1, 2012, you may read freely available joke books on Google books. On April 1, 2012, if I am still living, you may tell me a joke that you like. Such a rule set may not do anything very impressive, but it also isn’t going to cause a catastrophe.

4. Maybe trying to give a general, rule-based moral theory is the problem. Have you considered how moral particularism might be implemented for an AI? What about virtue ethics? (I didn’t see either of these in your impressive list of citations in “The Singularity and Machine Intelligence,” but I might have missed something.)

  (Quote)

Luke Muehlhauser January 21, 2012 at 10:24 am

Jonathan Livengood,

Thank you for your comments. I basically agree with all of them, I just don’t have the space in either the blog post or the paper to make every appropriate qualification and clarification. The paper has been submitted for Springer’s ‘The Singularity Hypothesis’, but it is only a draft; I’ll have another chance to revise it. I may be able to add some footnotes that hint toward the kinds of clarifications you suggest.

  (Quote)

Hunt January 23, 2012 at 4:05 am

Perhaps this is a naive question, but wouldn’t one of the fist things altered by a recursively self-improving intelligence be it’s own value system? And if the response to this is that it will be possible to engineer a value system that prevents the desire to fundamentally self-alter value systems, isn’t that an unjustified assumption?

  (Quote)

Stephen R. Diamond January 23, 2012 at 9:59 pm

“This is also why moral philosophers have spent thousands of years failing to find a simple set of principles that, if enacted, would create a world we want.”

[Quote from "Value is Complex."]

Which moral philosophers have tried to solve that problem? The only doctrine I’m aware of that assumes it’s possible to “implement” a moral theory is that of the utopian socialists (“From each according to his abilities, to each according to his needs.”)

Artificial intelligence, you’d think, has different constraints than philosophies moralizing about humans just because researchers can “implement” freely.

  (Quote)

Stephen R. Diamond January 23, 2012 at 10:37 pm

You give the impression (which I expect you did not intend to give) in both “Value is Complex” and also in “The Singularity” that for most (all?) decision problems, there is an outcome that all humans would prefer.

This denial of any ultimate conflicts of interest is central to the moral theories of Richard Carrier; they’ve gotten some support around here. They rest, it seems ultimately, on an anti-Darwinian group selectionism. At least that’s the most plausible explanation I can think of for assuming that values would evolve coordinated with all other people’s values. I think most “lay Darwinians” think in terms of group selection–so its influence isn’t adventitious.

  (Quote)

antiplastic January 24, 2012 at 11:11 am

All this carrying-on about group selection really is a lot of silly talk and a good example of the faith-based science worship pomos and religionists alike love to accuse atheists of. (It doesn’t help at all that this is taking place under the umbrella of Yudkowsky’s creepy Lesswrongian cult.)

“Basing” morality on group selection has two fatal flaws.

First, it is bad science. There is no such thing as group selection, or at least, no clear and agreed-upon real world example of it. Second, it applies to genes, not people. Any morally sane person understands that the summum bonum in life is not cranking out as many babies as possible before you die.

  (Quote)

Jonathan Livengood January 25, 2012 at 1:09 pm

Stephen and antiplastic,

I think you will find that Yudkowsky is as critical of group selection as it is possible to be in light of the mathematics of population genetics. That is, he denies that group selection has large effects — being swamped out by selection on individuals, though he acknowledges that some multi-level selection appears in the mathematical models.

As to my comment to Luke, I was making a point about decision theory. I don’t see the relevance of group selection one way or the other.

  (Quote)

Stephen R. Diamond January 25, 2012 at 3:13 pm

As to my comment to Luke, I was making a point about decision theory. I don’t see the relevance of group selection one way or the other.

From what I’ve seen, too, Yudkowsky isn’t a group selectionist.

On the relevance of group selection–you had written, “You give the impression (which I expect you did not intend to give) in both ‘Value is Complex’ and also in ‘The Singularity’ that for most (all?) decision problems, there is an outcome that all humans would prefer.”

Decision theory provides no basis for the existence of an outcome that all humans would prefer, but it seems germane to point out that evolutionary biology doesn’t provide the necessary constraint either, whereas group selectionism (were it applicable) might supply it. Perhaps I was too preoccupied with group selectionism on account of another thread, but group-selectionist thinking is connected in the popular mind with the idea that morality (assuming it is an adaptation) can represent universal interest.

  (Quote)

Stephen R. Diamond January 28, 2012 at 12:43 pm

3. I am very skeptical of the claim that there are no safe wishes smaller than an entire human value system. Let me give an example that supposes your Golem Genie. Suppose I say to the Golem, “You are allowed to do only the following two things. From now until April 1, 2012, you may read freely available joke books on Google books. On April 1, 2012, if I am still living, you may tell me a joke that you like. Such a rule set may not do anything very impressive, but it also isn’t going to cause a catastrophe.

I can see how it might cause a catastrophe. The Golem interprets “freely available” as anything it can hack into and joke as anything a human would find funny (including ironic).

But there’s an ambiguity. What’s a safe wish? The claim that any isolated wish is potentially corruptible may be true, but the likelihood of corruption may be small.

All said, it’s hard to see how an AI proponent can concede so much, yet stay in the business.

  (Quote)

Leave a Comment