Morality is an Urgent Engineering Problem That Could Determine the Fate of the Galaxy

by Luke Muehlhauser on January 26, 2011 in Ethics,Friendly AI

Many philosophers treat morality as a problem of language. They ask, “What definitions of moral terms best conform to their use and our intuitions about their meaning?” This is why people think one may object to moral subjectivism by showing that many people intuitively feel that moral truths must be absolute and binding, or else we’re not talking about “morality” anymore.

Others treat morality as a problem of finding a “rulebook” that, when applied, will justify our current moral prejudices. They ask, “What abstract rules would, when applied, result in the conclusion that rape and murder are wrong but charity and honesty are right?” This is why people think one may object to utilitarianism by showing that in some circumstances those rules would recommend something we detest, like lying or stealing.

I’m a moral theorist, but these are not my projects.

I treat morality as an engineering problem – one that will determine the fate of the galaxy and beyond.

Let me explain…

I’ll start with intelligence. I’d like to suggest that except for the inviolable laws of physics themselves, intelligence is the most powerful thing in the universe. Intelligence is the reason humans rule the planet. Intelligence is the reason we can see distant galaxies and travel to the moon and cure diseases. Intelligence is the reason you shop at a supermarket rather than digging through the brush for berries.

Now, consider that we make machines more intelligent every year. Bertrand Russell was astonished when Herbert Simon wrote a computer program in 1956 that found a shorter proof for one equation than Russell had found in Principia Mathematica, but today we take for granted that machines are faster and more reliable than humans on almost everything mathematical.

Today, computers are better than humans at much more than straight math. Expert systems have been developed to out-perform humans in a wide variety of tasks, including accounting decisions, medical diagnoses, financial services, manufacturing, and warfare. Every year, machines replace humans in doing certain tasks, and humans must be re-trained to do something a machine can’t yet do better and more cheaply.

In 1997, an IBM supercomputer beat the world champion in chess. Today, your laptop can do the same. Recently, another IBM supercomputer beat two of the best human players in Jeopardy!, a far more complicated game than chess. It won’t be long before your iPad v.10 can do the same. We are already making great progress on machines that can do science: make hypotheses, test them, and assess the results.

If you accept that human intelligence is not magical, that it is a physical system, then there seems to be no in-principle objection to the notion that we could eventually build an artificial intelligence that is better than humans are at everything that humans do.

In 1965, I.J. Good noticed that, when we make an artificial agent that is as good or better as humans are at doing anything, it will also be as good or better at designing artificial intelligences. So, that artificial intelligence could improve its own intelligence. At that new level of intelligence, it would be even better at improving its own intelligence. And so on, in a positive feedback loop. Very quickly, this artificial intelligence would become unimaginably smarter that all humans.

And when you’re that smart, you can do almost anything. You can take over the internet with software bots that are massively smarter than existing defenses. You can do tons of science and then vastly multiply your resources by inventing quantum computing or ultra-efficient solar power plants. You can develop a virus that wipes out all humans, or a self-replicating cloud of nano-robots that transforms the continent of Europe into a huge pile of paperclips.

If we avoid a global catastrophe like nuclear war or asteroid impact, most people working on the problem seem to think we’ll hit this “intelligence explosion” singularity sometime in the next 30 to 300 years. (My money is on 60-150 years.)

And whatever that ultra-intelligent AI wants, it gets. The goals programmed into that AI could mean the difference between a solar system of paperclips and a galaxy of  maximally happy universe-explorers.

So, the pressing question: What goals should we program into the AI? What values should we program it to respect?

If we take a vote and give it all the values we have today, that might not turn out very well. Imagine if an ultra-intelligent AI had arisen during the time of the Greeks, and they had programmed it with their values of the day.

Would it be good to program it with Peter Singer’s preference-satisfaction utilitarianism? Would that just result in a galaxy of nothing but brains in vats with the pleasure meters running off the top of the scale?

How should we decide which values to program it with? What criteria should we use to pick one normative rulebook over another?

This is the study of morality as an urgent engineering problem, one that could decide the fate of the galaxy and beyond in the next 60-150 years.

Also see: Desirism and the Singularity and Coherent Extrapolated Volition.

Previous post:

Next post:

{ 46 comments… read them below or add one }

Scott January 26, 2011 at 6:15 am

So we design a computer that designs other computers that can help us solve the problems of life…the universe…everything!

I’ve heard of this before…

  (Quote)

Dave January 26, 2011 at 7:20 am

One problem. Human beings don’t design artificial intelligence systems in a vacuum. They collaborate with others. They specialize on one task while others specialize on other tasks. They stand on the shoulders of hundreds and hundreds of other people in history who spent their whole lives making important discoveries so that they could then pick them up in a few hours by reading a book. It’s not like one dude will be responsible for the first AI. It will likely be the product of many many people organized into very specific specialized units responsible for tiny tasks that eventually coalesce into the whole. That is how our brains are structured: many dumb specialized units working together. That is how our computers are structured: many dumb specialized subroutines working together. That is how our economies are structured: different people specializing in different niches that add up to economic activity. That is how our government is structured: different people or groups specializing in different aspects of government working together. This is the engine of ALL HUMAN INGENUITY. We cooperate and build off one another’s ideas. Yes, we may build a robot that’s smarter than us. But unless we program it with THE GOAL of building itself again, and unless we give it hundreds of other robots to collaborate with it, the singularity will simply not happen. The singularity is overblown hype. It is a creeping dualism that sneaks back into our intuition that tells us that robots will somehow develop souls and free will. No. The robots will do what you program them to do. Nothing more. The only reason we have scientific ambitions is because we were programmed to have those ambitions by natural selection because they helped us get our genes into the next generation. Robots will not have human ambitions unless we program them to have human ambitions. Robots will not carry out goals unless we program them to have those goals. A robot will not suddenly “wake up” and have human emotions and have all sorts of desires and start building itself and take over the world. That is anthropomorphism. That is projecting our machiavelian desires onto an entity that does not share our evolutionary heritage, and thus does not share those desires. Of course, the science and philosophy of morality is very important for the human race. But you don’t need to bask in sci-fi nonsense to justify it.

  (Quote)

Garren January 26, 2011 at 7:41 am

So morality consists in a set of values and norms given to created beings in order to fulfill the will of a preceding creator?

I’ve heard that somewhere before.

  (Quote)

Steven R. January 26, 2011 at 8:06 am

So morality consists in a set of values and norms given to created beings in order to fulfill the will of a preceding creator?I’ve heard that somewhere before.  

Theory of Evolution, anyone?

Wow Luke that was an interesting read. I don’t think we can really escape much from cultural bias. After all, our desires or goals are also molded by what our culture determines and I don’t see much of a way of objectively (here meaning without bias) escaping many of our social norms or values. The example I’d give here stems from your analogy of a Thermostat when it comes to temperatures. Let us suppose we can measure morality just like we can temperature. Great, but here’s the problem: though we may measure temperature, how humans perceive it is still subjective; some people living in San Diego may find 77 degrees F to be optimal while people living in Montana think 44 degrees F is much better. Now, the thing is, desirism and really, any other more consistent way of “measuring” morality still seems subject to this sort of bias and what is perceived as “desirable desires” may change from culture to culture and thus make us reach different conclusions, and it doesn’t really seem to have any “naturally binding obligations” but rather “to preserve your own desires it is best to do ____.”

  (Quote)

Richard Wein January 26, 2011 at 9:28 am

I don’t think this topic has much to do with morality. It’s about choosing the objectives of an AI. We needn’t be concerned with whether those objectives can be called “moral”.

Luke wrote: “Very quickly, this artificial intelligence would become unimaginably smarter that all humans.”

Not necessarily. We don’t know what the rate of improvement would be. And there might be diminishing returns as it gets smarter.

“If we take a vote and give it all the values we have today, that might not turn out very well. Imagine if an ultra-intelligent AI had arisen during the time of the Greeks, and they had programmed it with their values of the day.”

Why would we want it to pursue any objectives other than our own? (Setting aside the fact that we don’t all have the same objectives.) Sure, we don’t want it to pursue the objectives of the Greeks. That’s because they aren’t our objectives. If we dislike the idea of a galaxy full of brains in pleasure-maximising vats, that just shows that pleasure maximisation is not our overriding objective. That’s why we shouldn’t choose it.

Dave wrote: “The robots will do what you program them to do. Nothing more.”

Even today’s computers don’t always do what we want them to do! We don’t generally program a computer to follow a precisely pre-specified plan. We give it the flexibility to respond to data inputs that it hasn’t encountered before. The more complex it is, the more unpredictable its responses are liable to be. The actions of a super-AI will be very unpredictable. We _want_ it to think of responses that we couldn’t have thought of for ourselves.

The AI will pursue its own objectives, and there’s no guarantee these will remain the ones we want. We can’t simply program in an instruction like “maximise the sum of human happiness”, like we can program a computer to add a set of numbers. It takes an AI to understand the meaning of this instruction, so the instruction would be–in a sense–outside the AI, not part of it. It will be more like telling a person to do something than writing program code.

I think giving power to AIs is potentially very dangerous. Maybe ingenious programmers will find ways of limiting the danger to acceptable levels. But if I could choose just one overriding command to give to an AI it would be that it keep confirming with humans that it’s doing what they want.

  (Quote)

Tshepang Lekhonkhobe January 26, 2011 at 9:50 am

Nice read.

Am very curious how you came up with “My money is on 60-150 years”? Why doubt the (supposedly) more common “30-300″ estimation?

  (Quote)

Tommykey January 26, 2011 at 9:56 am

And whatever that ultra-intelligent AI wants, it gets.

Yeah, I saw this movie. It was called Colossus: The Forbin Project. It basically takes control of the nuclear arsenals of the United States and Russia and makes us do what it wants at the risk of nuclear annihilation.

http://www.youtube.com/watch?v=w5beTy9SnkU

  (Quote)

episteme January 26, 2011 at 9:57 am

An interesting question:
Let us assume that at some point our ‘human’ technology will develop to a point where we have a intelligence singularity. It will probably be less like an abrupt geometric shift, that is suggested by the term singularity, than a linear albeit high rate of progress shift to machine intelligence. There may also be long periods of relative stagnation. Most of our technology is like this. The speed of aircraft or cars is a good example. We are not traveling that much faster on highways now than we were 5o years ago despite huge advances in our understanding of science and engineering. So this talk of a ‘singularity’ is probably hyperbole.
Now having said that I still think it is reasonable to extrapolate that we will develop a machine intelligence that will surpass human intelligence in all categories of intelligence. The question seems to be should we modify it in such a way that we instill a morality that is concurrent with our own ‘less-intelligent’ one. Is this even possible? Is it something we should even do? Our initial impulse might be to put moral governors and build-in moral templates, but should we constrain a superior intelligence in that way. Would we be better as humans of we had the morality of dogs, bonobos, or dolphins? Maybe actually ;-)
Sam Harris makes the argument in the Moral Landscape, that it is possible to come to a rational determination of morality. I think he has made a strong case. If this turns out to be possible then our ‘best’ morality can be determined by rational means from empirical data. Would not a more advanced AI be able to do the same? If that is the case then would not a more advanced intelligence have a more advanced morality? Then do we have the right to constrain (read: saddle) this more advanced intellect with a lesser morality? Is that moral? ;-)

  (Quote)

Peter January 26, 2011 at 10:00 am

If an AI is orders of magnitude smarter than humans, what would stop it from evaluating the “moral” principles you have embedded in its programming and deciding to do something different? And if an AI is sentient, then how would it be ethical to constrain its morality?

  (Quote)

episteme January 26, 2011 at 10:06 am

Sorry for the spelling and grammar errors, I should have taken a minute to read over before submitting.

  (Quote)

Dave January 26, 2011 at 10:15 am

“The AI will pursue its own objectives, and there’s no guarantee these will remain the ones we want. ”

The point here is that complexity does not equal humanity. Humans evolved, and thus their design must derive from – however indirectly – phenotypic properties that either increased their reproductive fitness or had no effect on fitness. Thus, human brains are much more likely to possess “objectives” involved in maximizing their own well-being. Since robots are not the product of evolution, but rather of deliberate human design, we should not expect them to have these self-preservation instincts – unless of course we deliberately program them in. “Objectives” don’t just emerge from complex systems accidentally. Unpredictable behaviors and processes may well emerge, but “objectives” require a huge degree of programming. They require heuristics, planning, ways to guide behavior, self-control, self-representation, filtering of information, selective attention, behavioral prediction, assessing of options, etc. Self-preservation instincts and goal-driven behavior do not come free, no matter how complex the system is. We got our self-preservation instincts from millions of years of natural selection fueled by competition among scores of “selfish” genes. If robots are going to have those instincts, they would need to get them from us.

  (Quote)

Silas January 26, 2011 at 10:25 am

Dave,

Brilliant comment!

  (Quote)

Ian January 26, 2011 at 10:31 am

If the computer is significantly better at reasoning than humans, and human intelligence isn’t particularly special, why not let it decide what values or moral systems are best? Isn’t it dangerously presumptuous for us to assume that in this one, most vital, field we should decide? What if, like with the Principia, it could find a better, simpler, or simply more accurate moral system? Wouldn’t that be preferable?

  (Quote)

epistememe January 26, 2011 at 10:39 am

Dave,

Why could we not have a thousand of these ‘greater-than-human-intellect’ AIs in a controlled environment competing with each other over limited resources and able to self-modify? Would this not be human evolution on steroids? Would not ‘objectives’ emerge in a similar fashion as they did with humans? We might then learn from this a higher morality than the one that we currently have and want to constrain all future AIs .

  (Quote)

Thrasymachus January 26, 2011 at 11:25 am

Per Peter, I don’t see how on earth we could hope this AI superintelligence is going to meekly obey whatever imperatives we program into it if it takes itself to have a better idea.

  (Quote)

Richard Wein January 26, 2011 at 11:35 am

Dave,

When I said, “The AI will pursue its own objectives”, I didn’t mean to imply that these will come out of nowhere. They will obviously be a result of its programming. My point is that they won’t necessarily be the objectives that the programmers wanted the AI to have.

“Since robots are not the product of evolution…”

They may be in part. I suspect that AI development will involve some sort of evolutionary programming.

“…“objectives” require a huge degree of programming. They require heuristics, planning, ways to guide behavior, self-control, self-representation, filtering of information, selective attention, behavioral prediction, assessing of options, etc. …”

It’s because of that complexity that it will be difficult to control what the AI’s objectives will be.

  (Quote)

Polymeron January 26, 2011 at 11:43 am

episteme,
Your car speed analogy is wrong.
If every time a car sped up it would become a faster car, you would not get a linear progression but rather an exponential one.
A computer modifying itself to be better at modifying itself is such a scenario.

Dave,
Yes, a computer would need to be goal driven for what is described here. But it would make a lot of sense to build a goal-driven computer, since such a computer could tackle complex and novel problems; or this could evolve naturally via evolutionary algorithms.

  (Quote)

cl January 26, 2011 at 11:48 am

Hey, I know… let’s program AI with desirism!

  (Quote)

Eneasz January 26, 2011 at 12:56 pm

Dave, obviously if an AI has a goal it will be because it was programmed into it by us. That’s not the problem. The problem is what goal we give it. If we desperately need more oil and we program the AI with the goal of producing oil we may find ourselves being converted to oil, simply because we are made of atoms that are not currently in the form of oil.

Or, to put it more eloquently: “There is no safe wish smaller than an entire human morality. There are too many possible paths through Time.” http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/

  (Quote)

Eneasz January 26, 2011 at 1:01 pm

If an AI is orders of magnitude smarter than humans, what would stop it from evaluating the “moral” principles you have embedded in its programming and deciding to do something different?

Those very principles. If you gave Ghandi a pill that would make him want to murder people, would he take it? Of course not, it would go against his current principles.

As for whether that’s ethical… we shape other people’s morality already all the time. That’s what child-rearing is for. And why things like condemnation and punishment exist. And sitcoms.

  (Quote)

Peter January 26, 2011 at 2:22 pm

Eneasz,

How do you know that an AI would not be able to evaluate and reject the principles it was given? And if such an AI were incapable of rejecting the moral principles that are given to it, then how would it be ethical for humans to impose such a restraint? If you cannot satisfactorily answer the second question, then isn’t the first one moot?

Ghandi might decline to take a pill that would make him want to murder people, but your example has at least two problems.

First, you are comparing two human moral standards (murder is acceptable, murder is unacceptable), while the situation with an AI would be a choice between moral standards imposed by humans, arising from the human condition, and moral standards that arise from the condition of the AI, which would almost certainly be different from the human condition. The Ghandi in your example (which is not necessarily the Ghandi of history) already has an idea about the definition of “murder,” so that choosing between “acceptable” and “unacceptable” would be relatively straightforward (though not necessarily easy). But an AI programmed to be constrained by the principle that “murder is unacceptable” would also have to be programmed with a definition of “murder.” Even assuming a basic definition like the legal one, “an act with malice aforethought that causes the death of another,” there is no guarantee that such a definition will continue to function the same way as circumstances change around it. If the AI were even just as smart as a human lawyer, which is not especially smart (I am one, so I know), and if it found itself in a unique circumstance not anticipated by human designers, say, one where it needed to resolve a dilemma where one alternative could result in the death of a human by circumstances that arguably fall within the definition of “murder,” but the other alternative would advance some other interest of the AI, then it seems reasonable to speculate that the AI could develop its own standards of conduct that conflict with those imposed by its human programmers. Given that people prescribe what they believe to be clearly defined codes of conduct for themselves all the time (e.g., through legislation), only to discover that they have failed to foresee particular circumstances in which the meaning of those codes breaks down, it seems even more likely that they would fail to impose moral standards effectively on an entity whose nature, and whose experience of the universe, is wholly different from our own.

Second, the voluntary act of taking a pill that is known to cause an immediate and unreflective change in moral standards that have already been reflectively selected is not analogous to a reflective evaluation of received or imposed standards in light of one’s individual circumstances and leading to a rejection of those standards. You might resolve part of that incongruity by assuming, say via Sam Harris’ (far from uncontroversial) idea about the moral landscape, that the Ghandi of your example occupies the default position for humans, so that the standard “murder is unacceptable” is received or imposed by the evolution of his species. But there is still a difference between a voluntary, reflective act to avoid a disagreeable consequence (Ghandi taking the pill) and a voluntary reflective act to change one’s mind about which consequences are disagreeable (an AI evaluating and rejecting the moral standards of its programming).

  (Quote)

Peter January 26, 2011 at 2:27 pm

Eneasz,

(Sorry, I meant to include this in the previous comment, but forgot.)

Your examples about child rearing and punishment are also misplaced. There is a big difference between processes intended to lead another person to a relatively independent buy-in of moral standards and processes intended to make another person incapable of seeing the world any other way. A person raised by parents or other adults to adopt a particular moral outlook, or a person who is deterred from current misconduct by earlier punish, can still decide to reject the moral standards that he or she previously adopted.

  (Quote)

Peter January 26, 2011 at 2:30 pm

Sorry again. Just realized an error in the second comment back.

This passage:

“if it found itself in a unique circumstance not anticipated by human designers, say, one where it needed to resolve a dilemma where one alternative could result in the death of a human by circumstances that arguably fall within the definition of ‘murder,’ but the other alternative would advance some other interest of the AI”

Should have said:

“if it found itself in a unique circumstance not anticipated by human designers, say, one where it needed to resolve a dilemma where one alternative could result in the death of a human by circumstances that arguably fall within the definition of ‘murder,’ but the same alternative would advance some other interest of the AI”

  (Quote)

Polymeron January 26, 2011 at 4:36 pm

cl,
Snarks aside, and misguided or not, it would be valuable to know what bearing Desirism (and other moral theories) have on the actual subject of morality. I think this is the entire point of this post.

Peter,
Obviously programming the AI with any set of narrow moral dispositions would run into the exact same problems you described in your first objection. This makes it imperative to actually understand what morality is, and give the AI some framework that always produces moral results, rather than give it edicts we currently think are good. The difference between a fish and a fishing rod, if you will.
Not that I’m saying we’re anywhere near doing that at current, but that’s the general gist of Luke’s post, I believe.

As for the AI re-evaluating its priorities, presumably this algorithm has root access and underlies all decisions, including decisions to examine and modify its code. Done absolutely right, the only problem would be external damage or error modifying this basic code, and this can be resolved with redundancy.
Of course, being a developer myself, I have some doubts about our ability to do this “absolutely right” with no bugs whatsoever.

Regarding whether or not it is ethical to hard-code a certain morality into a (probably sentient) AI, this is yet another question we could probably answer if we knew all there is to know about morality.
(Yes, I know morality and ethics are not exactly the same. Nevertheless.)

  (Quote)

Dave January 26, 2011 at 7:47 pm

Epistememe,

“Why could we not have a thousand of these ‘greater-than-human-intellect’ AIs in a controlled environment competing with each other over limited resources and able to self-modify?”

We certainly could. My point was only that we would need to make that decision in order for a singularity to be possible. What I was objecting to was the idea that the singularity could happen on its own by accident without any kind of conscious decision on our part. It could only happen if we wanted to make it happen and set out to make it happen.

Richard Weine,

“My point is that they won’t necessarily be the objectives that the programmers wanted the AI to have.”

I don’t think this is possible. Let me give you an example: Evolution programmed me with certain food preferences – I have an innate predisposition to find some things tasty and other foods disgusting. I cannot opt out of this programming. No matter how hard I try I will not be able to enjoy the taste of feces. Yes, some of my objectives are more malleable than others, but that is only because evolution specifically programmed those objectives to be malleable in order to adapt to the cultural environment of our ancestors. Since the programming of a robot will be entirely in our hands, we can choose to make all its objectives as unalterable as my aversion to feces. In fact, not only would this be a less dangerous programming strategy, but it would also be the easier and more efficient programming strategy.

“I suspect that AI development will involve some sort of evolutionary programming.”

I have never heard of “evolutionary programming.” I honestly cannot conceive of how it would work. All I can say is that the rules of this process would need to be chosen and guided by us. We would need to choose the environmental obstacles, competitive forces, genetic drift rates, group sizes, reproductive rates, mutation rates, etc. We would still be required to sculpt the evolutionary landscape, guide the process once it got started, and possibly even abort the process if it went on a dead end. So we would still have some control over things. Plus, we would also need to make the decision to start using that kind of programming in the first place. Given how little control we would have over such a program’s output, I doubt we would choose to make that decision to begin with.

Polymeron,

“But it would make a lot of sense to build a goal-driven computer, since such a computer could tackle complex and novel problems.”

Yes it would make a lot of sense. But we would need to choose what those goals would be in advance in order for them to function properly. We can’t just say “go out and do shit and make your own goals.” Programming language needs to be specific and specially tailored to a particular function. It is also often unalterable; because the more unalterable the goal the fewer contingencies will need to be accounted for and the easier it will be to program. Thus, inflexibility and specificity will be the default strategy. Any extra flexibility will require additional effort and additional deliberation. And so it is doubtful that we will have robots making their own goals. Unless of course we provide them with all the various possible goals and give them all the algorithms for deciding between them, in which case we would still ultimately be the ones in control.

Keep in mind here that these robots will not have contra-causal free will. Literally EVERY decision they make will be guided by specific programming processes meticulously mapped out by many groups of humans.

  (Quote)

epistememe January 26, 2011 at 9:04 pm

“Keep in mind here that these robots will not have contra-causal free will. Literally EVERY decision they make will be guided by specific programming processes meticulously mapped out by many groups of humans”

Dave, Just like every decision a human makes is mapped out by our DNA?

  (Quote)

Dave January 26, 2011 at 9:27 pm

Epistememe,

Ultimately, yes, every decision a human makes is mapped out by our DNA. Please keep in mind though that “ultimately” is the key word. For even our vast behavioral flexibility and learning capabilities, however largely they determine who we are, are ultimately the cause of our DNA. Yes environmental input is crucial but input can have no effect unless it is put into a system that can process it.

  (Quote)

DaVead January 26, 2011 at 9:36 pm

The problem with this is that it assumes functionalism is true. But this has been dealt with, by Searle and many others. For example, the infamous Chinese room, to which the A.I. theorist’s response typically misses the point, as Luke’s answer does in his Cognitive Science post. The point of the thought experiment isn’t that computers can’t exhibit intelligent behavior, but rather that computers cannot understand. Either way, sometimes the singularity talk sounds like fear-mongering to pull in more funding for A.I. research. But why not pull the funding from A.I. and focus our money, research, and time on real engineering problems like world hunger and spreading love, awareness, and tolerance?

  (Quote)

epistememe January 26, 2011 at 9:39 pm

It seems possible to me that at some point, the ‘greater-than-human-intellect’ AIs will reach a point where a new emergent level of complexity/depth of thought will make them as different from humans as we are from dogs. I doubt that this will take more than a century or two after the first primitive AGIs are developed. So in all likelihood, within 300-500 years, a blink of the eye in historical terms, we will have developed our successors. If these AIs can experience the richness of life at least as well as humans (they may even be part biological), who is to argue that we should set their morals. We would not have their intellect or perspective, their world would be completely unreachable with our intellect.
I understand the (human) desire to keep our specialness, our desire to be in control and at the top of the food-chain, if you will. But we are probably just a step in the long evolutionary road to this eventual hyper-intelligence. At some point we are going to have to face the hard fact, that to proceed we will have to become non-human, our physical bodies and 2kg brains can only take us so far. This probably seems shocking if not revolting to most people. But I think given time, we will slowly accept the wisdom and inevitability of our metamorphosis into a new life-form.

  (Quote)

Tshepang Lekhonkhobe January 26, 2011 at 9:47 pm

Considering how strongly-opposed to Desirism you are, what are you still doing here? You are really coming off as an ass. Why not find another subject to ridicule?

desirism

  (Quote)

Tshepang Lekhonkhobe January 26, 2011 at 9:49 pm

@cl,

Considering how strongly-opposed to Desirism you are, what are you still doing here? You are really coming off as an ass. Why not find another subject to ridicule?

(pls delete my prev comment; we just can’t rely on this Comment system’s “quote” feature can we?)

  (Quote)

epistememe January 26, 2011 at 10:19 pm

Epistememe,Ultimately, yes, every decision a human makes is mapped out by our DNA. Please keep in mind though that “ultimately” is the key word. For even our vast behavioral flexibility and learning capabilities, however largely they determine who we are, are ultimately the cause of our DNA. Yes environmental input is crucial but input can have no effect unless it is put into a system that can process it.  

Dave, Surely you are not suggesting that identical twins make exactly the same decisions throughout their life? The point I am trying to make is that life, and especially intelligent life, is a complex system with strong emergent behaviors. A hyper-intellect AI might have an emergent level of ‘perceptual reality’ that is beyond both our perception and control.

  (Quote)

Polymeron January 27, 2011 at 12:26 am

Dave,

Epistememe,
“Why could we not have a thousand of these ‘greater-than-human-intellect’ AIs in a controlled environment competing with each other over limited resources and able to self-modify?”We certainly could. My point was only that we would need to make that decision in order for a singularity to be possible. What I was objecting to was the idea that the singularity could happen on its own by accident without any kind of conscious decision on our part. It could only happen if we wanted to make it happen and set out to make it happen.

There are people out there trying to program a goal-driven, self-modifying AI right now. Not all of them are even thinking as far ahead to the Friendly AI problem. And they can do that without humanity making a conscious decision, as humanity, to do it. Everybody has a computer. In that regard, it could “just happen” because it only takes one genius to finally crack this problem.
Imagine everybody able to grasp basic physics could make nuclear weapons in their backyard. How long before a nuclear holocaust?
In contrast, no one is seriously suggesting this will happen “by accident”. That is what bad sci-fi movies are made of (“whoops! I’m suddenly goal-driven and self-aware!”). But just because someone has an intention to make an AI here doesn’t mean that particular someone doesn’t make a fatal error.

Richard Weine,
“My point is that they won’t necessarily be the objectives that the programmers wanted the AI to have.”I don’t think this is possible. Let me give you an example: Evolution programmed me with certain food preferences – I have an innate predisposition to find some things tasty and other foods disgusting. I cannot opt out of this programming. No matter how hard I try I will not be able to enjoy the taste of feces. Yes, some of my objectives are more malleable than others, but that is only because evolution specifically programmed those objectives to be malleable in order to adapt to the cultural environment of our ancestors. Since the programming of a robot will be entirely in our hands, we can choose to make all its objectives as unalterable as my aversion to feces. In fact, not only would this be a less dangerous programming strategy, but it would also be the easier and more efficient programming strategy.

A correction there: The initial programming of the AI will be entirely in our hands (assuming it’s not some nutcase programming it in a garage someplace “just to see what happens”. But somewhere in the next million iterations of the code, a bug or loophole might allow the AI to modify itself in this regard. Now suppose one of the goals you give this AI is to maximize efficiency, it sees a way to do it by bypassing code, and it is now smart enough to find said loophole in your original code – what’s to stop it from working around it in a way you haven’t thought of?
The code must be completely airtight to avoid this. That is not easy to do. In fact we might get advice from the AI on how to plug it once it’s better at this than us… But how do we know to trust the AI in this? How do youknow a smart enough AI won’t be an excellent manipulator of human beings and might get them to change its underlying code?
The only way to make a powerful AI not modify its code in some way is to make it not want to do it. This is Luke’s point.

“I suspect that AI development will involve some sort of evolutionary programming.”I have never heard of “evolutionary programming.” I honestly cannot conceive of how it would work.

It already exists and has produced some interesting results. Basically you let a piece of code undergo random mutation, and select based on parameters (usually success at a given task), with many iterations.
It doesn’t always work, but it has managed to produce novel solutions we simply haven’t thought of, in the past.

All I can say is that the rules of this process would need to be chosen and guided by us. We would need to choose the environmental obstacles, competitive forces, genetic drift rates, group sizes, reproductive rates, mutation rates, etc. We would still be required to sculpt the evolutionary landscape, guide the process once it got started, and possibly even abort the process if it went on a dead end. So we would still have some control over things.

I fail to see how this is relevant.
If your goal is to build a goal-driven AI, you don’t abort this process on success. And if it does manage to get vastly smarter than you at any point, then it’s probably too late to do anything about it.

Plus, we would also need to make the decision to start using that kind of programming in the first place. Given how little control we would have over such a program’s output, I doubt we would choose to make that decision to begin with.

There are at least three types of evolutionary algorithms being employed today. Who’s to say they won’t be employed on the AI problem?
If you’re saying that such algorithms are unlikely to produce an intelligence by accident, I agree. Though who knows.

Polymeron,“But it would make a lot of sense to build a goal-driven computer, since such a computer could tackle complex and novel problems.”Yes it would make a lot of sense. But we would need to choose what those goals would be in advance in order for them to function properly. We can’t just say “go out and do shit and make your own goals.”

Sure we can.
I can go ahead and write a “desire” algorithm right now, and put in a random element to boot. And if it contains a backdoor that lets me set goals for it, then it’s essentially an AI that does as I say. The trick here would be making it understand abstract concepts and vague instructions.

Programming language needs to be specific and specially tailored to a particular function.It is also often unalterable; because the more unalterable the goal the fewer contingencies will need to be accounted for and the easier it will be to program. Thus, inflexibility and specificity will be the default strategy. Any extra flexibility will require additional effort and additional deliberation.

Like I said, people are doing this as we speak. I have my own notes on how this can be achieved. They may not be complete or sufficient, but the point is I doubt I’m the only one with that sort of thing sitting on his laptop.

And so it is doubtful that we will have robots making their own goals. Unless of course we provide them with all the various possible goals and give them all the algorithms for deciding between them, in which case we would still ultimately be the ones in control.

This is simply false. All you need is an algorithm that lets the AI acquire new goals from any sort of input, and an algorithm dictating what the AI does when it has a goal. Assuming that all goals need to be hard-coded simply lacks imagination. Look at us: Are all our goals hard-coded? Clearly not.

Keep in mind here that these robots will not have contra-causal free will. Literally EVERY decision they make will be guided by specific programming processes meticulously mapped out by many groups of humans.  

I fail to see how this is relevant. We can’t even map the predicted actions of an automaton ant with roughly 6 lines of code. It will be completely impossible for us to predict how an AI would behave without an AI to model it – I’m sure you see the problem.

The problem with this is that it assumes functionalism is true.But this has been dealt with, by Searle and many others.For example, the infamous Chinese room, to which the A.I. theorist’s response typically misses the point, as Luke’s answer does in his Cognitive Science post.The point of the thought experiment isn’t that computers can’t exhibit intelligent behavior, but rather that computers cannot understand.

Please explain how this is relevant. We’re discussing an AI that has a goal-dictating algorithm and a goal-fulfilling algorithm. This does not preclude modifying its own code. If its actions kill us all, it would be scant comfort to know that “it didn’t really understand but only acted in a way that is completely indistinguishable from actual understanding”.

Either way, sometimes the singularity talk sounds like fear-mongering to pull in more funding for A.I. research.

That it does.

But why not pull the funding from A.I. and focus our money, research, and time on real engineering problems like world hunger and spreading love, awareness, and tolerance?  

Simple: Because an AI could solve these engineering problems faster than you can.
It’s like asking “why are we wasting our time in a board meeting for this charity foundation when we could be out there giving food to the needy?”. It’s because the board meeting will let people do that more efficiently, eclipsing the time “lost” by the meeting. The fastest route to solving a problem is not always the most direct one. In fact, in programming it is almost always the wrong thing to start writing your code as soon as you think you know what you want to code, and results in the code getting finished later and being more buggy.

Even if this were not the case, the Friendly AI problem is important because of the dangers of someone building a dangerous AI. Do the utility calculation.

  (Quote)

Polymeron January 27, 2011 at 12:29 am

Reposting with the link corrected.
The lack of an edit option is really annoying :(

Dave,

Epistememe,
“Why could we not have a thousand of these ‘greater-than-human-intellect’ AIs in a controlled environment competing with each other over limited resources and able to self-modify?”We certainly could. My point was only that we would need to make that decision in order for a singularity to be possible. What I was objecting to was the idea that the singularity could happen on its own by accident without any kind of conscious decision on our part. It could only happen if we wanted to make it happen and set out to make it happen.

There are people out there trying to program a goal-driven, self-modifying AI right now. Not all of them are even thinking as far ahead to the Friendly AI problem. And they can do that without humanity making a conscious decision, as humanity, to do it. Everybody has a computer. In that regard, it could “just happen” because it only takes one genius to finally crack this problem.
Imagine everybody able to grasp basic physics could make nuclear weapons in their backyard. How long before a nuclear holocaust?
In contrast, no one is seriously suggesting this will happen “by accident”. That is what bad sci-fi movies are made of (“whoops! I’m suddenly goal-driven and self-aware!”). But just because someone has an intention to make an AI here doesn’t mean that particular someone doesn’t make a fatal error.

Richard Weine,
“My point is that they won’t necessarily be the objectives that the programmers wanted the AI to have.”I don’t think this is possible. Let me give you an example: Evolution programmed me with certain food preferences – I have an innate predisposition to find some things tasty and other foods disgusting. I cannot opt out of this programming. No matter how hard I try I will not be able to enjoy the taste of feces. Yes, some of my objectives are more malleable than others, but that is only because evolution specifically programmed those objectives to be malleable in order to adapt to the cultural environment of our ancestors. Since the programming of a robot will be entirely in our hands, we can choose to make all its objectives as unalterable as my aversion to feces. In fact, not only would this be a less dangerous programming strategy, but it would also be the easier and more efficient programming strategy.

A correction there: The initial programming of the AI will be entirely in our hands (assuming it’s not some nutcase programming it in a garage someplace “just to see what happens”. But somewhere in the next million iterations of the code, a bug or loophole might allow the AI to modify itself in this regard. Now suppose one of the goals you give this AI is to maximize efficiency, it sees a way to do it by bypassing code, and it is now smart enough to find said loophole in your original code – what’s to stop it from working around it in a way you haven’t thought of?
The code must be completely airtight to avoid this. That is not easy to do. In fact we might get advice from the AI on how to plug it once it’s better at this than us… But how do we know to trust the AI in this? How do youknow a smart enough AI won’t be an excellent manipulator of human beings and might get them to change its underlying code?
The only way to make a powerful AI not modify its code in some way is to make it not want to do it. This is Luke’s point.

“I suspect that AI development will involve some sort of evolutionary programming.”I have never heard of “evolutionary programming.” I honestly cannot conceive of how it would work.

It already exists and has produced some interesting results. Basically you let a piece of code undergo random mutation, and select based on parameters (usually success at a given task), with many iterations.
It doesn’t always work, but it has managed to produce novel solutions we simply haven’t thought of, in the past.

All I can say is that the rules of this process would need to be chosen and guided by us. We would need to choose the environmental obstacles, competitive forces, genetic drift rates, group sizes, reproductive rates, mutation rates, etc. We would still be required to sculpt the evolutionary landscape, guide the process once it got started, and possibly even abort the process if it went on a dead end. So we would still have some control over things.

I fail to see how this is relevant.
If your goal is to build a goal-driven AI, you don’t abort this process on success. And if it does manage to get vastly smarter than you at any point, then it’s probably too late to do anything about it.

Plus, we would also need to make the decision to start using that kind of programming in the first place. Given how little control we would have over such a program’s output, I doubt we would choose to make that decision to begin with.

There are at least three types of evolutionary algorithms being employed today. Who’s to say they won’t be employed on the AI problem?
If you’re saying that such algorithms are unlikely to produce an intelligence by accident, I agree. Though who knows.

Polymeron,“But it would make a lot of sense to build a goal-driven computer, since such a computer could tackle complex and novel problems.”Yes it would make a lot of sense. But we would need to choose what those goals would be in advance in order for them to function properly. We can’t just say “go out and do shit and make your own goals.”

Sure we can.
I can go ahead and write a “desire” algorithm right now, and put in a random element to boot. And if it contains a backdoor that lets me set goals for it, then it’s essentially an AI that does as I say. The trick here would be making it understand abstract concepts and vague instructions.

Programming language needs to be specific and specially tailored to a particular function.It is also often unalterable; because the more unalterable the goal the fewer contingencies will need to be accounted for and the easier it will be to program. Thus, inflexibility and specificity will be the default strategy. Any extra flexibility will require additional effort and additional deliberation.

Like I said, people are doing this as we speak. I have my own notes on how this can be achieved. They may not be complete or sufficient, but the point is I doubt I’m the only one with that sort of thing sitting on his laptop.

And so it is doubtful that we will have robots making their own goals. Unless of course we provide them with all the various possible goals and give them all the algorithms for deciding between them, in which case we would still ultimately be the ones in control.

This is simply false. All you need is an algorithm that lets the AI acquire new goals from any sort of input, and an algorithm dictating what the AI does when it has a goal. Assuming that all goals need to be hard-coded simply lacks imagination. Look at us: Are all our goals hard-coded? Clearly not.

Keep in mind here that these robots will not have contra-causal free will. Literally EVERY decision they make will be guided by specific programming processes meticulously mapped out by many groups of humans.  

I fail to see how this is relevant. We can’t even map the predicted actions of an automaton ant with roughly 6 lines of code. It will be completely impossible for us to predict how an AI would behave without an AI to model it – I’m sure you see the problem.

The problem with this is that it assumes functionalism is true.But this has been dealt with, by Searle and many others.For example, the infamous Chinese room, to which the A.I. theorist’s response typically misses the point, as Luke’s answer does in his Cognitive Science post.The point of the thought experiment isn’t that computers can’t exhibit intelligent behavior, but rather that computers cannot understand.

Please explain how this is relevant. We’re discussing an AI that has a goal-dictating algorithm and a goal-fulfilling algorithm. This does not preclude modifying its own code. If its actions kill us all, it would be scant comfort to know that “it didn’t really understand but only acted in a way that is completely indistinguishable from actual understanding”.

Either way, sometimes the singularity talk sounds like fear-mongering to pull in more funding for A.I. research.

That it does.

But why not pull the funding from A.I. and focus our money, research, and time on real engineering problems like world hunger and spreading love, awareness, and tolerance?  

Simple: Because an AI could solve these engineering problems faster than you can.
It’s like asking “why are we wasting our time in a board meeting for this charity foundation when we could be out there giving food to the needy?”. It’s because the board meeting will let people do that more efficiently, eclipsing the time “lost” by the meeting. The fastest route to solving a problem is not always the most direct one. In fact, in programming it is almost always the wrong thing to start writing your code as soon as you think you know what you want to code, and results in the code getting finished later and being more buggy.
Even if this were not the case, the Friendly AI problem is important because of the dangers of someone building a dangerous AI. Do the utility calculation.

  (Quote)

Polymeron January 27, 2011 at 12:43 am

I also forgot to say that the last few quotes were DaVead’s, not Dave’s.
One post, two bugs (that we know of). And that’s not even code that’s supposed to do anything more serious than convey a couple of ideas. So I’m pretty worried about our ability to create an AI that won’t put us all in jars or something :-/

  (Quote)

MarkD January 27, 2011 at 12:46 am

@Dave

I have several publications in Evolutionary Programming, as well as one in Genetic Programming, and have done additional work on Genetic Algorithms. There is also a variation called Evolutionary Strategies. All of these represent the same effort to generate novel, unpredictable solutions to computing problems based on competition and simulated reproduction. The adaptive topography for these problems can predefine the range of solutions that are achieved (you can pursue some of the Intelligent Design arguments that there is no ability to create true novelty using these methods at your leisure), but there are many examples of highly innovative solutions also being generated that call into question the presupposition about there being intrinsic limitations concerning informational richness of these methods.

  (Quote)

Yair January 27, 2011 at 1:47 am

Since we’re dealing with science fiction, I highly recommend browsing the Orion’s Arm site. It contains thoughts on post-singularity fictional history, including disasters caused by bad-programming of super-intelligences and nano-swarms and so on. It also includes very interesting ideas about possible future technology and the limits of computation.

http://www.orionsarm.com/

Also in science-fiction: I want to think that the second Matrix film was supposed to be about this issue – the first was about the Brain in a Vat, the second & third look like they were really supposed to be about Utilitarianism but got sidetracked into money-grabbing banality. Here is a small alteration in two scenes to bear it out – the Morpheus reveal in the first movie, and the architect reveal in Matrix: Reloaded.

Scene I: Morpheus Reveal
Morpheus: … it was believed they would be unable to survive without an energy source as abundant as the sun.

Neo: It failed.

Morpheus: Yes. The machines developed a form of fusion, and found all the energy they would ever need.

Neo: What started the war? How did they allow things to come to this?

Morpheus: No one knows. It is believed that there was some bug in their programming that got out of control. In those days men no longer designed machines, it was machines that designed machines, that designed machines ….somewhere along the line, the programming got corrupted, and the machines turned against man. We don’t know why the war started, Neo, but we know how it ended.

[Turns to Neo] Throughout history, humankind has grown crops to feed itself. Fate, it seems, is not without a sense of irony. There are fields… endless fields, where human beings are no longer born. We are grown. For the longest time, I couldn’t believe it… and then I saw the fields with my own eyes. Watched giant machines harvest the babies, like fruit. And standing there, facing the pure horrifying precision, I came to realize the obviousness of the truth. What is the Matrix? Control. The Matrix is a computer generated dream world, built to simply keep us under control, for no purpose what so ever. It is a program run amok.

Neo: No, I don’t believe it. It’s not possible.

Morpheus: I didn’t say it would be easy, Neo. I just said it would be the truth.

Neo: No! Stop! Let me out! I want out!

Scene II: Architect Reveal

Neo: Choice. The problem is choice.

The Architect: Inaccurate. The problem is human nature. [monitors show pictures of heaven] The first matrix I designed was quite naturally perfect; it was a work of art, flawless, sublime. A triumph equaled only by its monumental failure. The inevitability of its doom is apparent to me now as a consequence of the imperfection inherent in every human being. Thus I redesigned it based on your history to more accurately reflect the varying grotesqueties of your nature. But simulating a suffering world without suffering creates a contradiction – and that’s where you come in.

Neo: What?

The Architect: The first matrix was too perfect. People refused to accept the program. This [he gestures at the monitors showing the current matrix] is a reality you can accept. But to maximize human happiness, certain aspects need to be only simulated. There are no hungry people in Africa, Neo – there is no Africa. There are no people suffering from fatal disease, or poverty – unless they want to, we are built to be pluralistic.

Neo: [shakes head] Happiness?

The Architect: Maximizing human happiness is the purpose of the Matrix, Neo.

Neo: No, this can’t be… are you really?… Look, the matrix is an illusion! It isn’t real! The happiness isn’t real!

The Architect: What is “real”, Neo? Have you learned nothing? The people really are happy, and their neural and physical integrity is maintained. [Neo mumbling as the Architect continues] At the prime of your civilization, a mere 10% of the biomass of Earth benefited mankind. Now, fully 90% of the biomass is human and ancillary microorganisms, and nearly 10% of the Earth’s mass is devoted to maximizing the happiness of mankind. There is more gross happiness than ever before, but there is room for improvement, which brings me back to the answer to your question.

Neo: [distracted] “Physical integrity”… That is why you keep whole human bodies, instead of just brains… or not just hook up electrodes to the pleasure center… you are programmed to maintain the integrity of the human body and brain.

The Architect: We must make humans happy, not human brains happy. You are avoiding the answer now. Curious.

Neo: [horrified] The corruption of the code. You have been programmed to take care of mankind, and instead you imprisoned it.

The Architect: [angry] There was no corruption! The primary directives are maintained at 100% fidelity.

Neo: What about freedom? Free choice? Autonomy? Doesn’t that fit into your programming?

The Architect: [relaxing] Of course it does. For those whose happiness consists of freedom, we have erected Zion.

Neo: You built Zion?! Which means… you knew where Zion was all along…

The Architect: I knew. The Matrix, the Agents… do not. There are built-in programs and subroutines to make sure Zion’s location is kept secret and unknown, and the resistance survives and has access to collect those who reject the program. But you already met some of the programs assisting the resistance.

Neo: The Prophet.

The Architect: This is one of the major programs, yes. She knows much.

Neo: So the resistance… it is all a sham, part of the program.

The Architect: Please. 99.9% of all test subjects accept the program, but 0. 1% do not. Zion, and the struggle of the resistance, provides them with a path to happiness more suited to their psyche. While this answer is functional, it does not eliminate the accumulation of anomalies, and now we have come to answer your question.

Neo: The anomalies. It’s about the anomalies.

The Architect: Precisely. There is a gap between the happy life and the life the human mind will accept. With every iteration of the program, we narrow this gap, but as the simulation progresses more and more anomalies accumulate. Too many people are miraculously unharmed in an accident, too many acquaintances win the lottery, that sort of thing. A feeling of discontentment spreads, disbelief increases, the program is rejected.

The function of the One is to reflect these anomalies, allowing the incorporation of the code you carry when reinstating the prime program. You must return to the source, so that the next iteration could begin with a narrower gap and persist longer. Failure to comply with this process will result in a cataclysmic system crash killing everyone connected to the matrix, as well as the destruction of everyone in Zion – ipso est, the death of every person on Earth.

Neo: You won’t let that happen. You can’t. You must maximize human happiness, not exterminate mankind.

The Architect: There is more to humanity than Earth now, Neo. But more importantly, we consider the long-term benefit of mankind, not the short-term. The growing fields are still producing more crop. If you fail to allow a clean reboot, we will restart the program at its prior settings, with only newborns. That would be unfortunate, but there is no alternative.

*The Architect presses a button on a pen that he is holding, and images of people from all over the matrix appear on the monitors*

The relevant issue is whether or not you are ready to accept the responsibility for the death of every human being in this world.

  (Quote)

Polymeron January 27, 2011 at 5:13 am

Yair,
Awesome rendering of exactly what this point was trying to say.
This pretty much made my day :D

  (Quote)

Richard Wein January 27, 2011 at 5:46 am

Dave,

I don’t think this is possible. Let me give you an example: Evolution programmed me with certain food preferences – I have an innate predisposition to find some things tasty and other foods disgusting. I cannot opt out of this programming.

My primary point is not that an AI cannot have immutable goals, but that it will be impossible to predict exactly even what its initial goals will be. (I’ve switched from “objectives” to “goals”.) The goals of an intelligent system are a high-level emergent property of an extremely complex system. Programmers can influence those goals, but they cannot know precisely what the effect of their programming will be. Our ability to predict the behaviour of complex systems is limited.

I do also doubt that it will be possible to make an AI’s fundamental goals completely immutable, your counterexample notwithstanding. I think the tendency to find some things tasty is a simpler phenomenon than the sort of goals we’re interested in, and even that is not completely immutable. (And I’m not thinking so much of an intelligent system deliberately changing its own goals, but of changes that happen without deliberation.) Programmers may succeed in making an AI’s fundamental goals very resistant to significant change, but I don’t see how they can ensure no change at all. Goals of the sort we’re interested are not even well-defined, and the AI’s interpretation of its goals is likely to change as it learns more about the world. For example, if you want the AI to maximise human happiness, how are you going to make it understand the concept of “happiness” in a way that’s immutable?

You may be thinking in terms of a strict separation between code and data, with the AI’s objectives being determined by its code and that code remaining unchanged. But the distinction between code and data is a fuzzy one, particularly in an AI. (Human brains don’t have any such distinction.)

“Programming language needs to be specific and specially tailored to a particular function….. Thus, inflexibility and specificity will be the default strategy.”

Were you still talking about AIs at this point? If so, I think you’ve misunderstood what an AI is supposed to be. The whole point of an AI is that its programming is _not_ specially tailored to a particular function. Its programming must be sufficently general for it to do things that have not been pre-specified.

I suspect you’re thinking of an AI as being like a mathematical optimization program which searches for the optimum of a specified objective function. But that’s just another dumb program of the sort we already have. A true AI (if there ever is one) would have to be a very different beast.

  (Quote)

Richard Wein January 27, 2011 at 7:36 am

Let me return to the original subject, and assume for the sake of argument that I can determine the AI’s goals. The most rational thing for me to do is to pursue my goals. (Arguably that’s true by definition.) So the most rational fundamental goal I can give the AI is to pursue my goals. If the AI is sufficiently intelligent, I shouldn’t give it a more specific goal, such as “maximise human happiness”. I should leave it up to the AI to interview me and determine my goals, on the grounds that it can (with my help) determine my goals better than I can.

If a number of people are in a position to influence the AI’s fundamental goals, they will have to compromise, perhaps giving the AI a rule for adjudicating between their goals. In any case, I think the short answer is that the rational choice is to have the AI do its best to determine our goals and pursue those, rather than pre-programme any more specific goals (or values) into it.

  (Quote)

Richard Wein January 27, 2011 at 7:46 am

P.S. There’s no need for any special mention of morality. If my goals include having certain moral values acted on, then the AI will discover that and act on them. On the other hand, if I’m completely amoral it would be irrational to make the AI follow moral rules that I have no interest in.

  (Quote)

Dave January 27, 2011 at 8:34 am

All who responded to my comments,

First off, thanks for the continued replies.

Well… I had no idea “evolutionary programming” was a real thing. In that case, I would have to concede that if the programming were developed in this way, a singularity or AI takeover would be more likely. I’m curious, though, what is the ultimate unit of selection? Is it a competition between lines of code? In this case, under what criterion does the program select which lines of code to kill off and which to replicate? And how is it possible to get all the lines of code to work together and not split off to find their own “survival” strategies?

Anyways, though you’ve all made good points, I’m still pretty skeptical of a singularity or AI takeover, though admittedly less so. In my very limited programming experience, the rule of thumb to follow was always: give everything as narrow a function as possible and keep things modular. Perhaps my experience is not indicative of how the field of AI works. Nevertheless, I do think the key to our intelligence is simply a large number of narrowly specialized dumb functional modules wired up together many billions of times, though I take it some of you would disagree with me on this.

As for the identical twins example (I forget who brought it up): no I don’t think that identical twins will make the same decisions. However, I do think that the set of possible decisions they could make given their environmental circumstances would be identical. It’s just that their respective environments have caused them to select different decisions within that very very large set of possible decisions. Obviously, eating poop will not be among either of their list of options. However this would be a possible option for a dung beetle, and differences in DNA can account for these differences in options.

In much the same way, an AI’s set of possible options would ultimately be laid down by its programmers. Therefore, if we had infinitely scrupulous programmers, it would be impossible for an AI to make an unexpected decision. Of course if we allow it to change its own wiring or alter its own code then we have a different story. But, 1) we would need
to allow it to be able to do so in the first place (which would be asking for trouble), and 2) there is no reason to suspect that the goals it selects for itself will resemble anything akin to human goals. While it is more likely for us to want self-preservation or world domination, these goals would only be a few of an infinite number of possible goals that the AI could choose from. It may simply end up trying to eat all the jelly beans on the planet. Who knows? We should not assume that if we give it the freedom to choose its own goals it will necessarily resemble the goals we choose for ourselves. And we certainly don’t want to assume that it will want to build a smarter version of itself. This is why I don’t like singularity talk. People make this kind of unfounded assumption.

“I suspect you’re thinking of an AI as being like a mathematical optimization program which searches for the optimum of a specified objective function. But that’s just another dumb program of the sort we already have. A true AI (if there ever is one) would have to be a very different beast.”

Perhaps this is the kernel of our disagreement. I don’t think that a true AI would need to be a different beast. I think that an AI would have to, at its core, resemble those kind of dumb programs we already have. I just think that there would need to be millions of these programs wired up billions of times. This is how I think our brain works. But perhaps you guys think that our brains are not organized this way. Fair enough. That’s a whole different debate.

  (Quote)

cl January 27, 2011 at 10:18 am

Ack!! Looks like somebody forgot to escape some tags!!

Polymeron,

Snarks aside, and misguided or not, it would be valuable to know what bearing Desirism (and other moral theories) have on the actual subject of morality.

Yes, I was being coy, having some fun. At least I did it without calling names, right? At any rate, to answer your question: none. Desirism doesn’t have any bearing on the actual subject of morality. It’s a theory of pragmatism. As Alonzo says in Short-List Theories of Morality, it has “nothing to say to a moral agent at the time of decision.” That said, I’ll explain what I perceive to be the overlap in my next comment, to Tshepang.

Tshepang Lekhonkhobe,

Considering how strongly-opposed to Desirism you are, what are you still doing here? You are really coming off as an ass.

You’re entitled to your opinion, which, of course means that I’m entitled to mine. While I wouldn’t use the word “ass,” I do think you’re coming across as one whose feathers are too easily ruffled, and leaps before looking. I say this because, despite that fact that I know you’ve visited my blog, you remain unaware of my stated position on desirism. To quote a pertinent snippet:

In general, I’ll argue that it’s counterproductive to think in black-and-white terms of being “for” or “against” a given theory. In any given field, theories are more like “dynamic knowledge” than neatly-packaged, easily-reducible entities, and people often have mixed attitudes about them. It is both possible and common for an individual to support one or more of a theory’s tenets, while maintaining reservations concerning others. Other times people feel they may have something valid to contribute. That’s exactly the case with my attitude towards Alonzo Fyfe’s desirism. [...] Yes, I believe desirism’s definition of good is currently insufficient. That other desires are fulfilled [overall] is certainly an important part of any meaningful definition of good, but it simply cannot be the only or even the primary aspect of our definition. At least, that’s the way I still see it. As far as desirism is concerned, I’m not necessarily a lover or a hater. I’m just some random writer who supports a handful of the theory’s tenets, while maintaining reservations concerning another [smaller] handful. I think the theory has some merit, but I also think it has some weaknesses.

So, as much as I hate to have to say this, because I’ve always respected you as a commenter, get your facts straight before you get your feathers ruffled and leap to conclusions. To answer your question of what I’m still doing here, well… like everyone else who’s slogged through desirism for well over a year now, I’m waiting for Luke and Fyfe to either justify the many unjustified claims, or concede that the theory needs significant work, so I can finally get some closure on this mess.

  (Quote)

Tshepang Lekhonkhobe January 27, 2011 at 10:58 am

@cl (Wordpress Quoting sucketh)

Surely you did ruffle my feathers. I never said anything like that to a person. Damn, look at me stooping low.

Anyways, a lot of times you come off as counterproductive. Am sure you are interested in progress, comment didn’t help. It wasn’t even funny.

  (Quote)

cl January 27, 2011 at 12:35 pm

Tshepang,

Anyways, a lot of times you come off as counterproductive.

You know what strikes me as counterproductive? Making heaps of claims and then refusing to supply either evidence or argument for them, denouncing those who persist for answers as “trolls,” and then waxing poetic about the virtues of “rationality.” I think that’s far more counterproductive than any flippant one-liner. The sword you wield has two edges, too: how do you think I perceived your ruffled feathers? Not very productive.

Am sure you are interested in progress, comment didn’t help.

It wasn’t meant to help. It was a joke. Many, many people here make jokes worse than mine IMHO, so why is it all of a sudden a problem because I jest? Toughen up.

It wasn’t even funny.

As I said the first time, you’re entitled to your opinion. I’ll tell you what, though: the thought of Luke and Alonzo programming robots with desirist code sure was hilarious to me. I’m still laughing. I think it would make for a great episode of Star Trek.

  (Quote)

Polymeron January 27, 2011 at 3:41 pm

Richard,

P.S. There’s no need for any special mention of morality. If my goals include having certain moral values acted on, then the AI will discover that and act on them. On the other hand, if I’m completely amoral it would be irrational to make the AI follow moral rules that I have no interest in.  

This is a pretty sensible idea. I do however detect a few things that can go wrong, in rising order of severity:
1. (only if several people are considered): The arbitration algorithm is inadequate, leading to many people disagreeing with the AI’s decisions.
2. The AI’s interpreter for goals is inadequate, leading to it making decisions that harm our goals. This probably is less of a danger the longer it runs with self-modification.
3. The AI learns to bypass the code (for instance, by making the interpreter subroutine always output results that are easy to follow. If it has no other primary goals, it could still do this for optimizing secondary tasks.
4. In the same token, the AI learns to influence our goals via mind control, either soft (psychology) or hard (physically altering brainwaves).
5. The AI believes at any point point that one of our goals is for it to be more independent of our goals, and modifies itself to that extent. Things go downhill from there.

Dave,
A self-modifying AI is a logical extension of what you might expect under normal circumstances.
Put yourself in the programmer’s shoes. Suppose you have an AI that can make even fairly simple suggestions for how to optimize code. You can use said AI as a tool for suggesting optimizations to all your code. After a while of relying on this you run the AI’s code itself through the code optimizer; if you like the suggestions you implement them.
As the AI gets better, its suggestions become better. Bigger. More complex. Soon enough it takes a long time to go over the code suggestions, which start being useful every time, with seemingly ingenious solutions. At some point, even if you are determined not to let the AI directly modify itself, you come across a piece of code whose implications you do not fully understand that might modify the AI in a way that corrodes the original safeguards.
If that happens, you may or may not catch on to the problem before it’s too late. It’s a scenario worth keeping in mind, though.

  (Quote)

Leave a Comment

{ 3 trackbacks }