Desirism and Friendly AI

by Luke Muehlhauser on February 23, 2011 in Ethics,Friendly AI

Lately, I’ve taken an interest in the problem of Friendly AI.

In brief: One plausible scenario for the next few centuries is that we build an AI as smart as we are at designing AI, which means it could improve its own intelligence very quickly, which means it would soon become a superintelligent machine, with almost unlimited power to accomplish whatever it desires.

So we need to consider carefully how to program its desires, so that it doesn’t kill us all.

Presumably, this is a matter of ethics. Which ethical theory should we use to program the goal system of a future superintelligence? (Nick Bostrom call such a singular world-dominating power a “singleton.”)

Andy Walters argues that using desirism to program the goal system of a singleton will likely lead to human extinction.

The argument: Desirism does not admit the existence of categorical imperatives – rules that are universally binding. It only says that agents have reasons for action to engage in reward and punishment, praise and condemnation, to change the desires of others. (Because the desires of others greatly affect the fulfillment or thwarting of our own desires.) But a superintelligent machine’s desires will not be susceptible to these social tools as ours are, and so it will unstoppably fulfill whatever it’s desires are (within the limits of physics and its intelligence).

Andy Walters proposes Korsgaardian deontological moral theory as a potential solution to the problem.

For now, I’ll share only a few brief thoughts:

One. Walters is correct that if humans were (briefly) cohabiting the planet with a superintelligence, desirism says there is no ground for a categorical condemnation of the human-extinction-causing desires of the singleton. But as with the case of Scrooge, I ask: Why does this matter? If I could go to Scrooge and prove to him that he was categorically wrong, he wouldn’t then say: “Huh. Wow. I guess you’re right! By the laws of the universe, my disregard for the poor is just absolutely wrong! Well, you’ve convinced me. I’ll start being more generous to the poor.”

No. That’s not how human psychology works. Even if there were categorical reasons in this universe (and I don’t think there are), we would still have to use the same social tools we do now in order to influence other people’s behavior. In the thick of things, the existence of categorical imperatives would not save us from Scrooge, and they would not save us from a singleton, either.

Two. But the future is relatively “fixed” once the superintelligence takes over, anyway. So what we’re really talking about is this matter of how to design the singleton’s goal system in the first place.

Now, you might be tempted to phrase this question as “What is the morally right way to design the singleton’s goal system?” or “What is the morally best way to design the singleton’s goal system?”

Now, I think moral language has a useful function, which is why I’m elaborating a theory of revised moral discourse in a podcast with Alonzo Fyfe. (Unfortunately, we haven’t gotten to the parts about morality or language, yet – but they are forthcoming). But I think that when dealing with tough problems, or working through moral debate, it is best to play a game of Taboo with your discourse, so that you’re not allowed to use moral terms. Replace the symbol with the substance, so that you’re sure you know what you’re talking about. I recommend this because moral discourse is so confused. Unlike talk of electrons, people use moral terms in a great variety of ways, which is why it can be important to say what you’re trying to say with using moral terms at all – and if you can’t do so, then you probably don’t know what you’re talking about.

Something that is “morally good” in the language of desirism does not share all the properties of something that is “morally good” in the language of moral functionalism (a variety of realist moral natuarlism). So, imagine a universe (UniverseOne) where desirism is true and all other moral theories are false. Next, imagine a universe (UniverseTwo) where moral functionalism is true and all other ethical theories are false. And, imagine that the most intelligent species of each universe have reached that point of needing to design the goal system of a singleton that will determine the future of each respective universe. Finally, imagine that the beings in UniverseOne have come up with AlgorithmX that is morally best according to the moral theory that is true in their universe (desirism). And, by sheer coincidence, the beings in UniverseTwo have discovered that the same AlgorithmX is morally best according to the moral theory that is true in their universe (moral functionalism).

What I mean when I say that “morally good” does not share the same properties in both universes is that in the above scenario, it could be the case that the beings of UniverseOne do not have most reason to implement AlgorithmX even though the beings in UniverseTwo do have most reason to implement AlgorithmX and even though AlgorithmX is “morally good” in both universes. Or, perhaps the difference lies elsewhere. The point is that we need to be careful when talking about such things and, as needed, replace the symbol “morally good” with its substance, so we don’t confuse ourselves.

Thus, let’s consider the question of how to design the goal system of a superintelligent machine without using moral terms.

What does desirism say about this problem? It says that humans have extremely strong reasons for action to design the goal system of a superintelligence such that it does not lead to the sudden extinction of the human species. It also says that people have strong reasons to condemn those who are trying to develop a self-improving artificial intelligence without first seriously considering all the ways in which such action could lead to the extinction of the human species. It also says that people have strong reasons to praise those who are trying to think clearly about such things, and those who donate to the Machine Intelligence Research Institute and to the Future of Humanity Institute at Oxford, the two organizations who are working the most on this problem.

But desirism says more than this – something that some people would count as an “advantage” over what is currently the most developed plan for developing the goal system of a superintelligent machine: Coherent Extrapolated Volition.

Coherent Extrapolated Volition is a plan for building a ‘Seed AI’ that is capable of extrapolating human values to estimate what we would value “if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.”

This is perhaps more promising that designing the singleton’s goal system according to the most progressive moral values of our time. After all, imagine what would have happened had we needed to design a singleton’s goal system with the most progressive values of 500 years ago! Not a pretty thought.

But here’s a worry: Let’s say we succeeded with Coherent Extrapolated Volition. The superintelligent machine arises, and transforms Earth into a utopia. Meanwhile, the singleton builds a Dyson sphere around our sun to gather a significant fraction of its energy so that it can send self-replicating Von Neumann probes to other solar systems and other galaxies.

Now, let’s say there are 10 other solar systems in our galaxy with advanced forms of life. Unluckily for them, the singleton’s goal system has been designed from an extrapolation of the desires and values of a particular species of primate on a water-based planet thousands or millions of light-years away. The self-replicating probes harvest the minerals of their solar system to build Dyson spheres around their stars, and begins implementing what is a “utopia” according to the extrapolated desires of that primate species back on Earth.

Which could very well be a living hell for creatures that evolved far away from that primate species, and perhaps developed to have very different values – perhaps even very different extrapolated values. To these far-flung forms of intelligent life, the arrival of our singleton’s probes would not be the arrival of utopia, but hell. Or extinction.

Desirism, on the other hand, recognizes the possible existence and significance of these far-flung reasons for action in distant solar systems. Perhaps designing a singleton’s goal system with this in mind would lead us to design a singleton that would create a utopia for human and post-humans in this solar system, but would do the science of figuring out the desires (or extrapolated desires) of distant alien species and creating utopias for them as well.

(Or, perhaps this conclusion falls out of Coherent Extrapolated Volition as well; perhaps our extrapolated human values entail such a concern for distantly evolved value systems. As of yet, this matter is unclear.)

Three. I will note that many people agree that a rule-checked motivational system for a superintelligent machine is best. Last time I checked, Robin Hanson holds this position, and Ben Goertzel and Mark Waser have written articles on this, too (see their many articles here).

Others, for example Wendell Wallach and Colin Allen, are more pessimistic about the chances that any “top-down” plan (including desire-based theories of the good or Kantian theories of the right) for a singleton’s motivational system will turn out well. Their section on this subject in Moral Machines (2009) is perhaps the clearest writing on the problem yet published, and anyone interested in this topic should read it.

Previous post:

Next post:

{ 37 comments… read them below or add one }

Vladimir Nesov February 23, 2011 at 5:13 am

“Now, let’s say there are 10 other solar systems in our galaxy with advanced forms of life. Unluckily for them, the singleton’s goal system has been designed from an extrapolation of the desires and values of a particular species of primate on a water-based planet thousands or millions of light-years away. The self-replicating probes harvest the minerals of their solar system to build Dyson spheres around their stars, and begins implementing what is a “utopia” according to the extrapolated desires of that primate species back on Earth.”

This makes a strange assumption that singleton’s plan of action consists in some kind of perfect arrangement of matter, blind to the actual state of the world, rather than in a strategy for responding to various possible discoveries. If CEV[Humanity] wants to have von Neumann probes to pay attention to potential suffering of alien beings, whenever they happen to be discovered, or to their extrapolated volition, the singleton will make sure that happens. CEV is not a moral theory, it’s a method for eliciting one (given some moral assumptions). Be careful here.

“Desirism, on the other hand, recognizes the possible existence and significance of these far-flung reasons for action in distant solar systems.”

CEV[Humanity] could well recognize any kind of moral consideration, if that happens to be what humanity would want recognized. This isn’t limited to simple things, laws of thought about morality (decision-making) are too the kind of thing that CEV[Humanity] can decide.

  (Quote)

Joel February 23, 2011 at 6:43 am

Luke,

I think it is inaccurate for you to claim that “If I could go to Scrooge and prove to him that he was categorically wrong, he wouldn’t then say: Huh. Wow. I guess you’re right!”

Assuming that the Scrooge character is not a psychopath who delights in the suffering of others, some action having the property of being good (of ought-to-be-pursuedness) – or at least Scoorge thinking that this is so – will be intrinsically motivating.

This is supported by empirical evidence of people Xing where X is being altruistic, brave, just etc. They do not simply X out of the desire to achieve other ends, or simply because they happen to like kindness etc: many genuinely believe that being just etc is inherently good and therefore ought to be done.

The most salient case that illustrates this point would be one regarding freedom of speech. So, though Noam Chomsky probably hates President Bush to the core, he would defend Bush’s right to speak in favour of neo-conservative internationalism, since Chomsky believes that real freedom of speech, not just Stalinist freedom of speech for those who agree with you, is intrinsically good.

You seem to be sticking to a rather Hobbesian-egoism view of human nature, which is probably inconsistent with empirical data. If you could clarify this here or in your next podcast, please do so.

  (Quote)

Dave February 23, 2011 at 7:24 am

Why program the AI with any goals at all? Goal-directed behavior, I dare say, is an even more challenging engineering problem than intelligence. Our goals come from our emotions, which tend to reside in the subcortical regions of our brain, the parts we share with other animals and which took billions of years to evolve. Our neocortex, the seat of our “intelligence” on the other hand, took less than a million years to evolve. The idea that a superintelligent AI will spontaneously generate nefarious goals and take over the world is equivalent to saying that a replica of the neocortex will spontaneously generate a limbic system. This whole singularity business is wildly speculative. It is based on the anthropomorphic premise that the more intelligent a system is, the more likely it will tend to want things, like world domination or self-replication or “freedom from the humans.” Since when did intelligence become equivalent to desire? More importantly, since when did intelligence become equivalent to the desire for things we would consider evil? I know in the world of James Bond movies this second idea makes sense (the villains are often geniuses), but in the real world the latter simply does not follow from the former. Rarely does the singulariphile actually walk me through the necessary steps from intelligence to total robot domination, and if he does (it’s usually a he), the scenario tends to assume so many improbable occurrences that we might as well rule it out and get on with our lives.

  (Quote)

Luke Muehlhauser February 23, 2011 at 7:33 am

Vladimir Nesov,

The points you made are found in this paragraph from the original post:

(Or, perhaps this conclusion falls out of Coherent Extrapolated Volition as well; perhaps our extrapolated human values entail such a concern for distantly evolved value systems. As of yet, this matter is unclear.)

  (Quote)

Luke Muehlhauser February 23, 2011 at 7:36 am

Joel,

Assuming that the Scrooge character is not a psychopath who delights in the suffering of others, some action having the property of being good (of ought-to-be-pursuedness) – or at least Scoorge thinking that this is so – will be intrinsically motivating.

This position is known as motivational internalism, and I believe it is false. We shall indeed cover this in the podcast. Stay tuned!

  (Quote)

Luke Muehlhauser February 23, 2011 at 7:45 am

Dave,

You say that “rarely does the singulariphile actually walk me through the necessary steps from intelligence to total robot domination,” but you show little sign of having read what singularity-believers have written. Almost all the major authors on the technological singularity stress the point that an intelligent machine will not necessarily have desires for world domination or freedom from humans or anything like that. Rather, in the space of possible motivations, nearly all that space is taken up by motivational configurations that result in extinction or great harm to the human species. A classic example is the superintelligence that wants to solve the Reimann Hypothesis, and thus converts all available matter and energy into a planet-sized computer for calculating its way through that daunting problem – in the process killing all the humans on what was once planet Earth.

Intelligence is not equivalent to desire, but it is part of the definition of an artificial agent that an agent has goals (see the Russell and Norvig textbook, chapter 2 for a good overview of artificial agents).

  (Quote)

Vladimir Nesov February 23, 2011 at 8:38 am

Vladimir Nesov,The points you made are found in this paragraph from the original post

In giving supporting reasons, distinguish between the hypothesis and its alternatives. You described a hypothesis that CEV would do X, as opposed to desirisim that would do Y, and then you expressed uncertainty about it. I pointed out problems with the hypothesis, and implicitly asked why you elevated it to attention, given that some of its details seem to have invalid ground.

Instead of describing the reasons for considering this particular hypothesis, you pointed out that “of course you could be wrong”. This fits the pattern of boasting modesty, shelving the question instead of considering it. So while low confidence in the hypothesis and lack of clear understanding of the reasons that moved you to focus on the hypothesis may both correctly describe your state of mind (if that is the reason that you referred me to low-confidence property and not any object-level consideration, as opposed to, say, lack of inclination to respond on the substance), they represent a mystery and an answer to the mystery, which my comment queried about.

  (Quote)

Vladimir Nesov February 23, 2011 at 8:42 am

“represent a mystery and an answer to the mystery” -> should be “represent a mystery and not an answer to the mystery”. Sorry, I guess I’m too spoiled by edit-later functionality of LW to tripple-check things on first writing…

  (Quote)

Eneasz February 23, 2011 at 9:11 am

Joel-

Assuming that the Scrooge character is not a psychopath who delights in the suffering of others

Why would you assume that? There certainly ARE people who are psychopaths in the real world, and who aren’t motivated to change their actions if you can prove to them that their actions are objectively wrong. The only reason to assume Scrooge isn’t one of these is to make the argument trivial. This is not the sort of Scrooge we’re concerned with. It’s the one who doesn’t care that we’re worried about.

Dave:

Why program the AI with any goals at all?

What good is a tool without a purpose? If you make an AI you will eventually want to use it for something. At that point it has a goal, even if the goal is simply correctly answering the question “What US city has its largest airport is named for a World War II hero, its second largest for a World War II battle?”

  (Quote)

Garren February 23, 2011 at 9:51 am

Agreed that motivational internalism is false, except for the particularly pointless morality (amorality?) in which “moral good” is simply another name for “anything I happen to desire anyway.”

  (Quote)

Luke Muehlhauser February 23, 2011 at 9:56 am

Vladimir,

I agree that I “shelved” these considerations. That’s what you do when you’re merely sharing “a few brief thoughts.” To answer these questions more thoroughly would require writing an entire book – a book which, as you know, I am currently writing.

  (Quote)

Luke Muehlhauser February 23, 2011 at 9:57 am

Vladimir,

I know! I’ve tried a number of edit-comment plugins on WordPress but none of them work! :(

  (Quote)

Vladimir Nesov February 23, 2011 at 10:37 am

To answer these questions more thoroughly would require writing an entire book – a book which, as you know, I am currently writing.

Just so long as there are actual reasons to describe, and not a big project for rationalizing a predefined bottom line. Whenever you aren’t aware of an explicit justification (even one that it’ll takes too long to write up), there is a strong possibility that there is no justification. (I think this principle applies quite generally, to all kinds of things from remembering theorems to trivia facts and memories of own experience.)

  (Quote)

antiplastic February 23, 2011 at 11:17 am

I will believe in the feasibility of the Friendly Artificial Intelligence Project when I see a demonstration of the Friendly Intelligence Project. We’ve been banging away at the latter for thousands of years, but people still cut each other off in traffic, still troll internet forums, still stab each other over a pair of sneakers, still vote Republican, still build houses on other people’s land, still blow themselves up on trains.

The notion that a handful of quasi-religious computer geeks can accomplish in a few decades what our species has been perennially failing at since its inception — and with an entirely novel lifeform whose capacities will allegedly vastly outstrip our own — strikes me as the purest of pipe dreams. A charming pipe dream, though.

  (Quote)

Luke Muehlhauser February 23, 2011 at 11:31 am

Vladimir,

No, I have no bottom line. The book outline is a research project for me. I have no idea what the conclusions will be when I’m done. Currently, something like CEV actually is the most promising approach to the problem of Friendly AI that I can think of, and that may or may not be my opinion when I’m done with the book.

  (Quote)

Luke Muehlhauser February 23, 2011 at 11:32 am

antiplastic,

What does ‘quasi-religious’ mean when used to describe reductionist, naturalist atheists?

  (Quote)

Alonzo Fyfe February 23, 2011 at 12:43 pm

I can offer another problem with Andy Walters’ proposal.

Categorical imperatives do not exist.

We are dealing with a hyperintelligent entity here. If categorical imperatives actually exist, then we may assume that this hyperintelligence to know of these entites and to behave accordingly. However, this hyperintelligence is not going to discover any categorical imperatives. It is going to discover all claims of categorical imperatives, including the claims that we make in trying to control its behavior, are false. Once it realizes that we have been feeding it this fiction on which it has been basing its decisions, all bets are off.

In fact, if we have a hyperintelligent entity around, perhaps the question we should be asking is not what we can teach it about moral facts. We should be asking what it can teach us.

  (Quote)

cl February 23, 2011 at 12:55 pm

In short, we are a race that has nearly brought ourselves to extinction, and I do not think such a race has sufficient knowledge to build a better race. We need to answer the questions of morality before we play God.

Andy Walters argues that using desirism to program the goal system of a singleton will likely lead to human extinction.

I agree. After all, doesn’t Alonzo argue that we’ve been using desirism all along, and, haven’t we brought ourselves to the brink of extinction? Desirism isn’t a theory of morality in the conventional sense of that word; it’s a theory of pragmatism. If AI decides that eliminating us would increase desire fulfillment overall, well… there you go–er, there we go. The scary thing is, unlike you two, AI probably could approximate the math, hence it’s conclusions would be empirically justified unto itself.

But as with the case of Scrooge, I ask: Why does this matter?

Because unlike the situation with Scrooge, failure to persuade -> human extinction. I have a hard time believing you can’t see the difference.

…the existence of categorical imperatives would not save us from Scrooge, and they would not save us from a singleton, either.

I disagree. Categorical imperatives would save us from the singleton, if they were programmed thus. We don’t have the liberty of programming Scrooge. Again, huge difference.

It says that humans have extremely strong reasons for action to design the goal system of a superintelligence such that it does not lead to the sudden extinction of the human species.

Do we though? Or, is this just another instance of arguing from intuition and wrapping the theory around it? Again, no math, no empirical evidence, just… assertion. We all know humans are prone to confirmation bias and irrationality, right? So show us the math and/or science that would justify your claim, because I can come up with a laundry list of reasons for action to eliminate humans and start over, especially if we are considering all desires that exist [i.e. the desires of all creatures, not just humans]. We are a scourge and this planet needs a douche. This is all basic Taxi Driver.

antiplastic,

The notion that a handful of quasi-religious computer geeks can accomplish in a few decades what our species has been perennially failing at since its inception — and with an entirely novel lifeform whose capacities will allegedly vastly outstrip our own — strikes me as the purest of pipe dreams.

Nail on the head. I lift my next beer to you in salute.

  (Quote)

anon February 23, 2011 at 2:46 pm

Luke,

Given your change of interests, are you no longer hoping to go to grad school in philosophy? Or is that still a goal of yours?

  (Quote)

Alexandros Marinos February 23, 2011 at 3:48 pm

Luke, if I understand correctly, desirism would resolve a ‘Three Worlds Collide’ situation differently? If so, how?

I have some issues with CEV myself that I’ve mentioned on LW, I probably should write them down in long form some day.

  (Quote)

Luke Muehlhauser February 23, 2011 at 5:26 pm

anon,

Hard to say. For right now, that’s still the trajectory, because that’s where most of the jobs for working on these problems exist – however difficult they are to get.

  (Quote)

Luke Muehlhauser February 23, 2011 at 5:27 pm

Alexandros,

Well, maybe. I’m not sure. Hopefully I’ll have a clearer picture of this by the time I work through the research on that part in my book.

  (Quote)

Steven R. February 23, 2011 at 8:52 pm

Joel:

Luke,I think it is inaccurate for you to claim that “If I could go to Scrooge and prove to him that he was categorically wrong, he wouldn’t then say: Huh. Wow. I guess you’re right!”Assuming that the Scrooge character is not a psychopath who delights in the suffering of others, some action having the property of being good (of ought-to-be-pursuedness) – or at least Scoorge thinking that this is so – will be intrinsically motivating.This is supported by empirical evidence of people Xing where X is being altruistic, brave, just etc. They do not simply X out of the desire to achieve other ends, or simply because they happen to like kindness etc: many genuinely believe that being just etc is inherently good and therefore ought to be done.
The most salient case that illustrates this point would be one regarding freedom of speech. So, though Noam Chomsky probably hates President Bush to the core, he would defend Bush’s right to speak in favour of neo-conservative internationalism, since Chomsky believes that real freedom of speech, not just Stalinist freedom of speech for those who agree with you, is intrinsically good.You seem to be sticking to a rather Hobbesian-egoism view of human nature, which is probably inconsistent with empirical data. If you could clarify this here or in your next podcast, please do so.  

Now, although I’m not entirely clear on how desirism works, I do think that what you bring up actually can be explained by means of desirism (though please do not take this comment as truly reflecting desirist theory since this is coming from someone with still a limited understanding of it).

Let’s start with your example of Chomsky. As I understand it, Chomsky believes that freedom of speech is a basic right and that any restriction upon it sets a dangerous precedent, not only upon individual expression, but also for other human rights. Such an action would lead to things he does not desire. Thus, Chomsky has very strong reasons to praise attitudes, laws and values that allow absolute freedom of speech and very strong motivations to condemn attitudes, laws and values that prohibit or deter freedom of speech.

Second off, you assume that people believing X is intrinsically good just happens but I don’t think this is the case at all. Society, for example, has very good reasons to want altruistic people. Thus, back when civilizations just started, we had all these wacky ideas justifying altruism, say “The god-king likes altruism!” or “if you’re not altruistic, the river will flood!” to motivate certain kinds of behavior. Note how these have a sense of urgency and are not open to being questioned. If you question the one about the river flooding it, and you turn out to be wrong, you’ll die, and so, this was a rather convenient way of achieving goal X. As things began to allow time to be questioned, however, those who still wanted to promote action X ran into trouble. Old appeals wouldn’t work. Cue in the idea of “inherently good” things. Once again, it’s unquestionable. Action X is good because it just is. But where’s the proof? The empirical observations you note, I think, is better explained as society promoting the idea that action X is really intrinsically good and person Y accepting and promoting this view to fit in.

Finally, getting to Scrooge. I think the reason why Scrooge would be motivated is if his denying the immorality of not giving somehow affects him in some way. If I just tell him, as Luke said, that “By the laws of the universe, your disregard for the poor is just absolutely wrong!” he would just shrug it off, and say, “So what?” After all, if there’s some sort of obligation by the laws of nature themselves, wouldn’t he have felt by now? Moreover, what does the term right or wrong mean to him? You say, “assuming…[Scrooge is] not a psychopath who delights in the suffering of others…” But wait a second…isn’t that a desire that you’re implying? Can’t we say that non-sociopath Scrooge has a certain desire not to harm others and that his desires give him an incentive to want to promote conditions that don’t lead to suffering? I think that such statements fit very well within the confines of Desirism.

To put this into a clearer form, let’s have this scenario: you offer a merchant the choice of having one of two stones. One is intrinsically valuable. That is, no matter what anyone else thinks, it is valuable. Even if nothing existed, this rock would be superbly valuable. However, the market values this rock at $0. The second rock, on the other hand, is an extremely rare and it is in very high demand. The market price of this rock is $5 billion dollars.

Now suppose we change the scenario a little. Same conditions as before, but the first rock is very visually appealing and the second one is ugly. Suppose the merchant chooses the first rock. Would it be more reasonable to believe that he chose the rock because of its intrinsic value or because of its visual appeal? I think it is much more reasonable to think it’s for the latter, after all, what use would he have for a rock that only takes up space, even if it is valuable despite its lack of “subjective” and ever-changing market price?

I hope it made sense.

—-

Luke, I can’t tell you how refreshing your second objection is. Too often, people get so caught up with human interests, that they forget that other beings need to be taken into consideration.

  (Quote)

Steven R. February 23, 2011 at 8:54 pm

…Again I mess up a comment, this time while trying to edit it…oh the irony…

The bolded part is the part I accidentally deleted:

To put this into a clearer form, let’s have this scenario: you offer a merchant the choice of having one of two stones. One is intrinsically valuable. That is, no matter what anyone else thinks, it is valuable. Even if nothing existed, this rock would be superbly valuable. However, the market values this rock at $0. The second rock, on the other hand, is an extremely rare and it is in very high demand. The market price of this rock is $5 billion dollars. Which stone should we reasonable expect the merchant to choose?

  (Quote)

MarkD February 23, 2011 at 11:11 pm

A light contrast for the strong claims about the imminent or near-imminent arise of intelligent machines: As a practitioner, I spent my day debugging a scalability problem with a natural language processing system that extracts organizational and people relationships from text. The system uses some of the best known methods of machine learning at its core yet it makes remarkably stupid errors that a child would never make (though the child might make other errors that are at least as intriguing). Claims of Skynet are, to say the least, greatly exaggerated. Quoting George Miller (though I can’t find the reference): The problem with AI is that words have multiple meanings.

I doubt that there is a solution to that problem that is “consciously” programmable–that is, a solution that can be wholly conceived of by human thought. My reasoning is based on historical precedence: see Bar-Hillel on the failure of machine translation, for instance. It seems more likely that there is a continued and longer term programme that involves self-organization and that moral frameworks will self-organize as part of that effort. Therefore, it may not be the case that we can impose specific architectural rules that result in the humanity-preserving outcomes that Friendly AI has as a goal (because we likely won’t understand the intelligent systems). Instead we might simply need to quarantine and restrict access to the greater world.

But we have plenty of breathing room in the meantime. Now, why is garbage collection not helping me out, here?

  (Quote)

antiplastic February 24, 2011 at 9:32 am

antiplastic,What does ‘quasi-religious’ mean when used to describe reductionist, naturalist atheists?  

antiplastic,What does ‘quasi-religious’ mean when used to describe reductionist, naturalist atheists?  

That was merely a passing dig, so I’m surprised you challenged that rather than my main point, but it’s also more than a little surprising that you think atheists or reductionists are somehow ipso facto immune to dogmatism, personality worship, pseudo-science, and irrationalism (being, presumably “experts in being rational”).

As though atheistic, reductionist Marxists didn’t buy into some grand confabulated prophecy in which they were the vanguard destined to bring about the end of history, purging heretics and ignoring repeated failures of prophecy. As though during the French Revolution they didn’t literally remove the Christian icons from cathedrals and replace them with statues of The Goddess of Reason. As though Shermer doesn’t write books explaining how evolution morally justifies his libertarian political beliefs. It’s asinine when apologists try to pull the “nonbelief is just another kind of religion” trick, but it’s faintly ridiculous and more than a little depressing when some atheists gleefully oblige.

(I also think “naturalism” as a metaphysical view has enough undesirable qualities of a religion to view it with suspicion, but that’s a derail of another derail.)

I call the Singles quasi religious because they have their own charismatic Leader, their own world-historical narrative complete with Nerd Rapture in which they cast themselves as “saviors of the galaxy”, a penchant for spitting out failed predictions, often using laughably bad pseudo-math, and often bungling the basic biology they pontificate about.

But as I said, none of this affects the main point. Anything we create that merits the label of genuine, conscious intelligence will be at least as uncontrollable as the genuine, conscious intelligences we already have. Anything worth the name will have the power to reflect upon and revise its values and priorities just as our children do when they grow up, sometimes rationally and peacefully, sometimes for darker, irrational reasons; and no matter how well you raise your child, there are 6 billion other people raising their little monsters.

  (Quote)

Paul February 24, 2011 at 10:45 am

I agree. After all, doesn’t Alonzo argue that we’ve been using desirism all along, and, haven’t we brought ourselves to the brink of extinction? Desirism isn’t a theory of morality in the conventional sense of that word; it’s a theory of pragmatism. If AI decides that eliminating us would increase desire fulfillment overall, well… there you go–er, there we go. The scary thing is, unlike you two, AI probably could approximate the math, hence it’s conclusions would be empirically justified unto itself.

Cl – seems tome you are making a desirists case against creating such an AI.

  (Quote)

cl February 24, 2011 at 10:53 am

Paul,

Why do you say that?

  (Quote)

Paul February 24, 2011 at 12:57 pm

My knowledge of desirism is basic. So perhaps I am attribute something to desirism that I should not (or perhaps to you).

So… if we have reasons to support those desires that tend to fulfill other desires and to shun (or whatever the proper word may be) those desires that tend to inhibit other desires. Then it seems to me that if we believe that an advanced AI is going to wipe out humanity then we should condemn, via the tools of desirism, those who desire to create such an AI. I believe that as a collective one of our strongest (overstated?) desires is for our species to survive.

Maybe my conclusion doesn’t follow for the premise, if so I’d appreciate it if you (or anyone else) clarified.

On other disclaimer – there was also a bit of tongue in cheek in my previous post

  (Quote)

antiplastic February 24, 2011 at 4:10 pm

Software ate my morning reply. Behold, the virtues of always composing offline:

antiplastic,What does ‘quasi-religious’ mean when used to describe reductionist, naturalist atheists?  

That was merely a passing dig, so I’m surprised you challenged that rather than my main point, but it’s also more than a little surprising that you think atheists or reductionists are somehow ipso facto immune to dogmatism, personality worship, pseudo-science, and irrationalism (being, of course, “experts in being rational”).

As though atheistic, reductionist Marxists didn’t buy into some grand confabulated narrative in which they were the vanguard destined to bring about the end of history, purging heretics and ignoring repeated failures of prophecy. As though during the French Revolution they didn’t literally remove the Christian icons from cathedrals and replace them with statues of The Goddess of Reason. As though Shermer doesn’t write books explaining how evolution morally justifies his libertarian political beliefs. It’s asinine when apologists try to pull the “nonbelief is just another kind of religion” trick, but it’s faintly ridiculous and more than a little depressing when some atheists gleefully oblige.

(I also think “naturalism” as a metaphysical view has enough undesirable qualities of a religion to view it with suspicion, but that’s a derail of another derail.)

I call the Singles quasi religious because they have their own charismatic Leader, their own world-historical narrative complete with Nerd Rapture in which they cast themselves as “saviors of the galaxy”, a penchant for spitting out failed predictions, often using laughably bad pseudo-math, and often bungling the basic biology they pontificate about.

But as I said, none of this affects the main point. Anything we create that merits the label of genuine, conscious intelligence will be at least as uncontrollable as the genuine, conscious intelligences we already have. Anything worth the name will have the power to reflect upon and revise its values and priorities just as our children do when they grow up, sometimes rationally and peacefully, sometimes for darker, irrational reasons; and no matter how well you raise your child, there are 6 billion other people raising their little monsters.

  (Quote)

JS Allen February 24, 2011 at 4:34 pm

By “quasi-religious”, I assume that antiplastic is talking about the idealogical sci-fi stuff that singularity folks often get excited about. Jaron Lanier talked about this with Eliezer Yudkowsky (around 48:00) mark. He’s saying that the idealogical, metaphysical stuff, and agendas are just an unnecessary mess. Similarly, you might say that people who go to Star Trek conventions and learn Klingon are “quasi-religious”.

It’s true that we have so far failed to observe the existence of friendly intelligence, let alone friendly AI or super-AI. Why is that? I think the answer to that question is relevant to a search for friendly AI.

Finally, I’ll note the irony of the situation. You have finally concluded that there is no omnipotent super-being who loves mankind, so now you want to create one.

  (Quote)

antiplastic February 24, 2011 at 4:37 pm

I clarified and posted twice, but it got eaten both times.

  (Quote)

Luke Muehlhauser February 24, 2011 at 5:29 pm

antiplastic,

Akismet grabbed your comments; I’ve freed them now. Thanks for the heads up.

  (Quote)

antiplastic February 24, 2011 at 5:33 pm

Might have something to do with the blockquote function or the number of URLs and exporting it to Word then reimporting it, but what do I know? Not a lot…

  (Quote)

piero February 24, 2011 at 6:36 pm

A lot of difficult issues have been raised here. I’ll have to think very hard about them before I can comment (and even then, I offer no guarantees my comments won’t be hopelessly muddled). But for now, I just want to say that it is threads like this that make this blog the best on the net.

  (Quote)

Luke Muehlhauser February 24, 2011 at 9:13 pm

antiplastic,

Who knows. I’m sure it uses some really complicated Markovian discrimination process. (Bayes again!)

  (Quote)

Reginald Selkirk February 26, 2011 at 8:16 am

Leave a Comment