Potential Causes of Total Human Extinction in the Next 200 Years

by Luke Muehlhauser on March 20, 2011 in General Atheism

Derek Parfit ended Reasons and Persons in this way:

I believe that if we destroy mankind, as we now can, this outcome will be much worse than most people think. Compare three outcomes:

(1) Peace.
(2) A nuclear war that kills 99% of the world’s existing population.
(3) A nuclear war that kills 100%.

(2) would be worse than (1), and (3) would be worse than (2). Which is the greater of these two differences? Most people believe that the greater difference is between (1) and (2). I believe that the difference between (2) and (3) is very much greater.

…The Earth will remain inhabitable for at least another billion years. Civilization began only a few thousand years ago. If we do not destroy mankind, these few thousand years may be only a tiny fraction of the whole of civilized human history.

The difference between (2) and (3) may thus be the difference between this tiny fraction and all of the rest of this history. If we compare this possible history to a day, what has occurred so far is only a fraction of a second.

Many share the view that total human extinction would be far worse than the loss of 99.999% of all humans, after which enough humans would survive that they could eventually repopulate Earth and perhaps the galaxy.

Moreover, total human extinction is, perhaps for the first time in recorded history, quite plausible. Experts who have voiced this concern include Martin Rees (Our Final Hour), John Leslie (The End of the World), and others (Global Catastrohpic Risks, from which most of the data below comes).

Perhaps we ought to take a moment of our time to assess these risks.

What disasters could cause total human extinction?

Well. The heat death of the universe definitely will. But that’s a long way off. Let’s be more precise.

What disasters could cause total human extinction in the next 200 years?

There is another reason for framing the question this way. If we can survive the next 200 years, we may by then be able to upload our minds into computers, make copies, and send them off in probes to thousands or millions of destinations in the galaxy.

Thus, it may be that these next 200 years are the most critical: when total human extinction is most plausible. This may even be the most critical time (for intelligent life) in the history of the galaxy. We haven’t received signals – or probes – from any other intelligent life in the galaxy (the Fermi paradox). Maybe they never existed because intelligent life so rarely evolves, or maybe all other advanced civilizations destroyed themselves before they got to the stage of uploading themselves to computers and sending out masses of probes.

So: what could cause total human extinction in the next 200 years?

Non-Prospects for Total Human Extinction

A volcanic super-eruption (VEI 8) happens on average every 50,000 years, the most recent being the Oruanui eruption in New Zealand 25,000 years ago. But even if a super-eruption occurred, humans are now such an adaptive species that even the largest super-eruption in Earth’s history would probably not kill off all humans. There would probably be hundreds of thousands – or millions – of us left to repopulate the globe.

Solar flares are never large enough to destroy our entire atmosphere, and thus our species.

A nearby supernova explosion might strip the Earth of its ozone layer, allowing penetration of UV rays. This would wipe out most of our species, but we are adaptable enough that we would (barely) survive underground. Likewise, a gamma ray burst aimed directly at Earth would kill most of us, but would spare some of those who happen to be underground at the time.

The Earth’s magnetic sphere disappears every few hundred thousand years as it reverses itself. But these events do not correlate with mass extinction events, and won’t wipe out humanity.

Climate change won’t happen quickly enough to kill more than a couple billion people in the next two centuries, even at worst.

A natural pandemic of the worst kind in Earth’s history would kill, at most, a couple billion people.

Nuclear war, even with future nuclear weapons even more powerful than today’s, would not kill at least a few dozen people who are deep underground in, for example, mines and data storage facilities and military bunkers.

Extremely Unlikely to Cause Total Human Extinction

Impact by a large asteroid or comet (> 25km in diameter) could wipe out the human species entirely. (A 10-15km asteroid impact in Mexico seems to have been what killed the dinosaurs.) If the rock is in a close-earth orbit, we’ll have decades or centuries of advance warning, but a comet or “dark Damondoid” wouldn’t given enough warning for us to react at all. Luckily, such events are far more rare than, volcanic super-eruptions, and extremely unlikely to occur in the next 200 years.

Biowarfare is pretty likely as cheap technology makes biology programmable, like a computer. A global and lethal airborn virus is unlikely to reach the most remote persons, including those deep underground. But a precisely engineered bioweapon could penetrate some strongholds protected even from, say, nuclear warfare, and thus I’ve graduated this risk to “extremely unlikely to cause total human extinction.”

Runaway physics experiments could, say, create a black hole that would swallow the Earth. Though worries about the Large Hadron Collider are unfounded, the general point remains that physicists are often creating forces and particles and environments that have never existed before on Earth, and playing with physical laws and dimensions we do not understand. The risk of runaway physics experiments could be nil or they could be somewhat high (a century from now, when our biggest physics experiments are even bigger). There is no cause for immediate concern, but because we have almost no idea what the likelihood of such disasters is, I’m graduating this disaster to “extremely unlikely to cause total human extinction.”

Merely “Unlikely” to Cause Total Human Extinction

Runaway nanotechnology could self-replicate beyond human control, taking up all resources (of a certain kind) in sight – perhaps filling the sky and even digging into the Earth and oceans. However, such intelligent self-replication may rely on another risk that is broader in scope, namely: unfriendly machine superintelligence.

Unfriendly machine superintelligence could wipe out humanity by converting all matter and energy in the nearby solar system into parts for achieving whatever goals were programmed into it (solving certain math problems or colonizing the galaxy or whatever). The path to such god-like powers for AI is this: Once an AI becomes as smart as we are at designing AI, it will be able to recursively improve its own intelligence, very quickly becoming vastly smarter and more powerful than any human resistance would be capable of stopping. Intelligence is the most powerful thing in the universe, and a superintelligent machine could easily destroy humanity unless it is programmed to preserve it.

Conclusions

I didn’t use to think so, but I’ve become persuaded that unfriendly machine superintelligence is the most plausible cause of total human extinction. If this is correct, then the single largest impact you can have with your charity dollars is to give all of them toward ensuring we develop artificial superintelligence that is friendly to human goals. Giving to stop global warming looks like a drop in a puddle in comparison.

Still, it is important to keep in mind that some risks that won’t cause total human extinction while still causing hundreds of millions of deaths may be more likely to occur than unfriendly machine superintelligence. Biowarfare and climate change examples; luckily, there are things we can do about them, too.

I must also admit that I am not an expert of human extinction risks. I invite correction. I don’t propose this post as providing a scholarly game plan, but as an invitation for more people to engage in researching this badly neglected subject: the possibility of total human extinction.

Previous post:

Next post:

{ 75 comments… read them below or add one }

Nige March 20, 2011 at 4:45 am

With regards to the danger posed by `runnaway physics experiments`, it is true that we cannot quantify all of the risks ( no doubt many of them are currently unknown ), but we can say that they are far too tiny to worry about. The collision energy of many cosmic rays on the surface of the moon and atop the Earth`s atmosphere is many orders of magnitude greater than those achievable by the LHC or indeed by any conceivable particle accelerator, yet these have been occuring continuously for billions of years and the Earth is still here! This catagory of possible risk to humanity can safe be safely dropped from your list!

  (Quote)

Paul Crowley March 20, 2011 at 5:06 am

First, climate change does present an existential risk: James Hansen for one warns of a “Venus Syndrome” we could not survive.

Second, I’m surprised you rate unfriendly machine superintelligence as “unlikely”. It seems to me that if we manage to avoid any other interruptions to “business as usual” in the next century or so, then superintelligence is very likely, and it seems pretty unlikely to be Friendly at the moment when Friendliness gets so little consideration.

  (Quote)

Scott March 20, 2011 at 8:04 am

2012? Where’s 2012 rank?

  (Quote)

Tarun March 20, 2011 at 9:00 am

Luke,

If you (and other singularitarians) think that unfriendly machine intelligence is the greatest threat to humanity, and you think that developing friendly AI is an extremely difficult problem, wouldn’t money be better spent on trying to prevent the development of a superintelligence in the first place rather than making sure that the superintelligence is friendly? Trying to solve the friendly AI problem seems like a much lower expected utility strategy than campaigning against the creation of machines with human level or greater intelligence. Why shouldn’t I be spending money and time on the latter goal?

  (Quote)

Bill Williams March 20, 2011 at 9:24 am

@ Luke said,

“the single largest impact you can have with your charity dollars is to give all of them toward ensuring we develop artificial superintelligence that is friendly to human goals. Giving to stop global warming looks like a drop in a puddle in comparison.”

Seriously? You’ve framed this issue as though total or near total extinction is the only issue to consider. What about the death of 100 million people from a war of machines? What about a very tiny loss of life that involves the brutal mutilation of your loved ones?

If we fund the development of friendly artificial superintelligence, we’ve also created unfriendly AI. It only takes a couple of assholes with turbans to fly a plane into a building. If the technology is developed, the villains on this planet will tweak it do their bidding.

So, based on this bullshit argument, you want us to fund the development of killing machines in the name of “charity.” This is bizarre as fucking hell.

  (Quote)

Jonatas Müller March 20, 2011 at 10:12 am

Most risks seem unlikely, and that’s good, and we should still prevent them. I think that human extinction by artificial intelligence is also quite unlikely and, especially, would not classify exactly as an extinction, but as a continuation of evolution of our civilization on Earth — as humans evolved from other primates, so would machines evolve from humans, and continue the process.

It would be preferable and more ethical to do a smooth transitioning and to maintain people, enhance them, make them happier, healthier and better, and not waste happy lives, and I don’t see what reasons an AI would have to unethically and purposefully kill humans. Furthermore it’s not difficult to completely restrain AI. Presumably it would not have capacity to directly alter its hardware, only software, if so. What could an immobile box do to conquer the world? It could be put on a simulated reality and not know about the real world, it could be monitored and restrained on speed, etc.

Unlike these measures, programming friendliness on AI seems futile in the long-term and biased. An AI would be able to freely explore philosophical topics, whether purposefully or incidentally while doing something else, and could discover ethics and how the friendliness that was inserted is against its ethical principles (in case it is). An AI that only does what’s programmed is a narrow AI, or no AI at all. A true AI would end up discarding its friendliness in case they don’t agree with it, or be friendly anyway in case they agree with it. In sum, it would act the way it would act regardless of any programmed friendliness. The programmed friendliness could only buy some time for the early maturation of AI.

Unless the development of AI is very careless, extinction from it should be an unlikely to very unlikely risk.

  (Quote)

Reginald Selkirk March 20, 2011 at 10:35 am

(2) would be worse than (1), and (3) would be worse than (2).

Only from a human-centric point of view.

  (Quote)

nige March 20, 2011 at 10:52 am

There are very many individuals and research groups all around the world working on issues relating to cognitive science and artificial intelligence, so our knowledge base is increasing steadily and the only way to prevent our eventual ability to create human level AI would be to permanently pull the plug on all this research. Clearly this is impossible. Even if some international body had the authority to pronounce such a ban, there would be plenty of individuals and research groups who would secretly ignore it. Research in this area is relatively inexpensive ( and so does not require the backing of a nation state ) and easy to conceal. Like it or not, human level AI will eventually be created.

A question those who imagine we will be able to control the behaviour of human ( or human plus ) level AI ( through the instillation of appropriate moral outlooks for example ) might ask themselves is – `what is there to stop belligerent or simply lunatic individuals or groups from doing just the opposite?` Once we know how to create human level AI the knowledge will be available to all who have the expertise to comprehend it, and implementing this knowledge in the creration of said AI will be open to vast numbers of individuals, research groups, companies and states at relatively little expense. It is absolutely certain that this knowledge will be multiply abused.

I`m afraid that providing our technological civilization continues, the eventual creating of human level AI, and `bad` AI at that, is absolutely inevitable. Luke is right to be extremely worried about this.

  (Quote)

Bill Williams March 20, 2011 at 11:23 am

Well stated Nige.

We were told that we needed nuclear weapons to make us safe. As indicated in Luke’s article, they are now the number one threat our very existence.

I agree with you that bad AI is inevitable (unless some other catastrophe strikes first). Anyone with pie-in-the-sky fantasies about how great life will be with superintelligence needs to remember that assholes will have it too. As if that isn’t bad enough, at some point even they may lose control.

Why would any sane person want to accelerate this process?

  (Quote)

Curt March 20, 2011 at 11:33 am

If a few people emerge from deep bunkers and mineshafts after an all out nuclear war what are they going to eat if they can dig themselves out from under the rubble?
We do not think we can really say how many people could die in a very bad pandemic.
Maybe we should not develope AI in the first place if it could one day kill us all. Yet I think that it will be developed becasue it could help save us from global warming or a pandemic BEFORE it kills us. I however will not give a penny to develope AI because I think that there will be more than enough people looking in to this to be able to make money.
We therefore will do what I say and that is to give money to KIVA and to anti war groups. You and you and you on the other hand can do what you can with who you can when you can.
Crunchy Crustacean Curt

  (Quote)

RadiVis March 20, 2011 at 11:52 am

It is not easy to restrain a superhuman AI. Even if it was restrained in some way, the main danger would come from humans wanting to free em, because of es superhuman intelligence, which could be used for all kinds of purposes – so why restrain em after all? In this context Eliezer Yudkowsky’s AI-Box experiments are interesting: http://yudkowsky.net/singularity/aibox
So, if there is superhuman intelligence, e will very likely be free very soon.

There’s also the problem that (ethically) unrestricted AIs probably will be more powerful than restricted ones. That may very well make survival of unaugmented humans unlikely. Still, cyborgs or uploaded humans who have reached superhuman intelligence may be in a position to resist unfriendly AI. I think we have a decent chance of survival, if we augment ourselves sufficiently (that is, by becoming beings which are effectively on par with superhuman-AI – some may say that’s not possible, but they don’t go far enough in their thinking. Extremely radical software/wetware/hardware upgrades could effectively turn us into superhuman-AI equivalent persons or group-minds). The merger of man and machine is safest way to a positive future. Here’s a related post of mine, which is slightly related to that topic: http://deathrant.net/2011/03/capability-augmentation-will-solve-all-our-problems/

Oh, and I like your analysis. It’s quite good. I’m just a bit skeptical whether survivors in underground facilities are able to rebuild a human civilization in any case. After a global catastrophe the world might look much more hostile than under normal circumstances.

  (Quote)

Nige March 20, 2011 at 12:03 pm

I only discovered this blog very recently, and have been challanged by much of what I have read here. In particular, I have greatly enjoyed the archive of philosophy podcasts, and hope that Luke can return to producing new ones in the near future.

As I said previously, it seems to me inevitable that should our technological civilization continue, we will eventually produce superintellignce, and some of this will quite deliberately be of a `bad` sort, which ( as Bill said ) will inevitably manipulate itself out of the control of its `masters` ( or indeed, be deliberately `set free` ). The results of all this will be quite appalling for the human race.

With the above in mind, I quite agree with Bill that I cannot understand why anyone would want to hasten the development of this technology ; we need as much time as possible to think carefully about all the issues petaining to it, in the hope that just maybe there is an as yet unforseen way to `contain` it. Personally however, I do not for one second think that such a thing will be possible; we are afterall talking about a superintelligence that would be able to out wit us, even if it wasn`t given more than a helping hand in this by beligerant, lunatic or ideologically driven humans.

As Jonatas commented, we might see our superintelligent creations as being our successors, and I am quite comfortable with this, providing we can pass on something of ourselves – including our `values` – to them, so there is no discontinuity in what most people would consider to be the most essential aspects of our humanity. Maybe in the end all we can do is try and create AI with a psychology that is to our liking, and hope that this version wins the inevitable conflict that will occur with its equally inevitable `bad` rival.

  (Quote)

JS Allen March 20, 2011 at 12:37 pm

It’s a fascinating topic. I think it all comes down to the probabilities you assign to different scenarios. Luke clearly said that global nuclear holocaust is a better outcome than an unfriendly singularity — and global nuclear holocaust would presumably hinder the birth of the singularity. So there must be some potential scenarios where investing in Armaggedon is more rational than investing in friendly AI.

For example, if there is a 99% chance that potential human investment can eternally constrain a super-intelligent singularity, but a 1% chance that our collective investment could produce an Armaggedon that would stall the singularity for at least 2,000 more years — would it be worth the existential gamble to invest in friendly AI rather than Armaggedon? Maybe. What if the odds were 30% in both cases? Maybe not, in that case. What if the odds were 2% and 80%? In that case, investing in Armaggedon might make a lot more sense.

Given the fact that the probabilities have such a huge impact on the investment decision, those probabilities should be based on more than wild speculation and wishful thinking. I’m not convinced that we know anywhere near enough to make a decision that’s better than flipping a coin.

In fact, we might ask whether it’s even possible to have a grasp of the requisite probabilities before it’s too late. Suppose that we sink all of mankind’s resources into friendly AI, and then realize that our faith was misfounded and that there is only a 1% chance of succeeding — and Armaggedon is no longer an option. We’ll put on sackcloth and ashes, and wail for the lost chance to provoke Armaggedon, but there will be no relief. Conversely, we could provoke several more Armaggedons over then next 20,000 years, and a future civilization may find that the friendly singularity was inevitable, and that our misery was pointless.

  (Quote)

Mark March 20, 2011 at 1:03 pm

Once an AI becomes as smart as we are at designing AI, it will be able to recursively improve its own intelligence, very quickly becoming vastly smarter and more powerful than any human resistance would be capable of stopping.

This is the central claim of transhumanists like Yudkowsky, and I’ve always found it puzzling. Maybe you can explain it to me. How does it follow from the fact that a human-level AI can begin recursively improve its own intelligence that it will grow “vastly” more intelligent than humans? Couldn’t its intelligence simply grow asymptotically towards some cap that, while technically superhuman, is not so vast as to be unstoppable?

  (Quote)

Alexander Kruel March 20, 2011 at 1:25 pm

Given the loss of 99.999% of all humans it is questionable if humans would ever be able to regain their old strength. Arguing that there is a chance is like arguing that someone who is brain dead might grow a new brain given future technological advances and turn into the same person again. If we focus too much of our resources on the possibility of artificial intelligence induced extinction we might miss the chance to mitigate other risks that are known to be factual. Your believe in the feasibility of artificial general intelligence capable of explosive recursive self-improvement is unjustified as it is merely based on extrapolations of our current understanding of the nature of intelligence and the possibility of efficient substrate neutrality, which is effectively nonexistent. We should gather more empirical evidence in support of this possibility before disregarding other problems that are already known to have happened and that will happen again.

  (Quote)

Luke Muehlhauser March 20, 2011 at 1:35 pm

Tarun,

I am indeed in favor of slowing progress in artificial general intelligence. But I don’t think total prevention can succeed. I explain why in the few pages of my book ‘Ethics and Superintelligence’ that are already written and available online.

  (Quote)

Luke Muehlhauser March 20, 2011 at 1:36 pm

Bill,

What you’ve suggested is the opposite of what I recommended. I think we should give to charity to fund the development of non-killing machines. Not killing machines.

  (Quote)

Luke Muehlhauser March 20, 2011 at 1:39 pm

Mark,

See Chalmers’ paper.

  (Quote)

Alexander Kruel March 20, 2011 at 1:47 pm

See my comment here for more on why I feel uncomfortable agreeing that mitigating unfriendly machine superintelligence should be our priority.

  (Quote)

Jonatas Müller March 20, 2011 at 2:41 pm

AI that isn’t narrow and limited to its programming would be much more powerful, and would be likely to freely explore philosophical matters, whether purposefully or incidentally while doing something else. Just as inserting friendliness would be futile for true AI in the long-term, so might be inserting evil, because the AI would discover its ethics and would somehow discard the dispositions that contradict it.

Should we worry about computers which just follow programming and are evil? Certainly true AI with their own ways of behavior, which we cannot force beforehand, would be more powerful.

  (Quote)

Curt March 20, 2011 at 2:52 pm

A lot of very interesting comments. I would give JS Allen a Gold Star and a smiley face.
I also think that Mark is on to something. The ability to think is one thing but the ability to act is quite another. If this AI has no means of gathering information it can not improve all by itself. If it has no means to communticate with its surroundings it can not cause any mischief.
It does seem plausible that at some future time if we are around long enough the one group of humans in a conflict with another group of humans would realease this machine from the chains that had been holding down to achieve a decisvie advantage over their advisary. Yet even then I doubt the a machine would turn deliberately malicious towards humans. Doubt is not certainty.
There are are paths that such a super intellegent machine could take. It could be paralyzed by indecsion. First of all such a mahine may decide that it wants to consider all relevant factors before making a decision it could then get stuck in an eternal information gathering mode. It may also be paralyzed by indecision because it can not forsee all of the consequences of its actions and with out fully understanding the consequences may decide to do nothing at all. Finally it may be paralyzed because even though it thinks it knows all the relevent facts and all of the knowable consequences it realizes that any choice involves a trade off of pros and cons, which naturally are based on values, (which a machine may not have) and it can not desicively say which out come it perfers. We humans may act in a similar pattern if it were not for our subconscience mind helping us to come to a decision.
Cumbersome Cockoo Curt

  (Quote)

Polymeron March 20, 2011 at 6:26 pm

Nige,
Something Luke hasn’t mentioned explicitly is it is assumed that the first superhuman AI would vastly outmatch any competing predecessors, due to having more resources (e.g. the other AI would not be able to consume the Internet which an AI already protects). Such an AI may well be able to regulate the dangers inherent in AI development better than us – IF it is Friendly. This provides the impetus for (responsibly) hastening, not slowing, the research on Friendly AI.

Jonatas Müller,
For clarity, Luke should have capitalized the term Friendly, which does not mean the AI will be chummy and helpful. Rather, it is a technical term in the AI research field that means that regardless of the number of iterations, the AI would not develop goals harmful to human well-being. In theory, this is to be achieved via making it understand the implications of its own reprogramming and desire to remain Friendly. In practice, we don’t (yet?) know a way to do it.

As Luke would probably be happy to reiterate, the problem is greatly exacerbated by our current inability to accurately define things like “well-being”.

  (Quote)

Rob March 20, 2011 at 6:41 pm

A philosophically serious consideration of the prospect of total human extinction has to contend with Benatar.

  (Quote)

Polymeron March 20, 2011 at 7:38 pm

Rob,

Depending on what you mean by “contend”, that may be somewhat difficult or very easy.

Evolutionarily, his opinions are irrelevant. People infected by these ideas both need to
fight an innate urge to the contrary, and bring no progeny they can teach at an impressionable young age. This means that there is little chance that these opinions would ever have any noticeable impact on actual policy, and so we can ignore them as a matter of staying practical. Barring the invention of moral calculus and him being correct, widespread consensus of these ideas is highly unlikely.

Then again, if by “contend” you mean that to be philosophically serious we need to closely examine the arguments, you could be right – except that I don’t think that this sort of philosophical debate actually generates answers.

“Hold on, let’s stop and think if maybe we want to go extinct” seems like a counter-productive approach. Any good reason why we should consider it? I don’t think philosophical completeness is a good reason or I’d be arguing with solipsists all day.

  (Quote)

JS Allen March 20, 2011 at 8:15 pm

Polymeron,

Benatar is one extreme, but it raises the point that Luke is essentially taking the other extreme. He’s implying that human existence is better than non-existence, no matter how miserable that existence is. That seems like a pretty remarkable stance for someone to take, if that person finds the “problem of evil” persuasive, or rejects the doctrine of hell.

People certainly can adopt that stance. For example, Alexander Pruss just argued that existence in hell is better than no existence at all. But that seems intuitively wrong to me, and not at all something I would expect an atheist to assume. FWIW, Katja Grace had an interesting post recently, showing that the talk about “good of existence” can be arbitrary.

  (Quote)

Polymeron March 20, 2011 at 8:27 pm

JS Allen,

[Luke]’s implying that human existence is better than non-existence, no matter how miserable that existence is.That seems like a pretty remarkable stance for someone to take, if that person finds the “problem of evil” persuasive, or rejects the doctrine of hell.

I would indeed see it as remarkable, except I don’t think this was at all implied. He says:

“Many share the view that total human extinction would be far worse than the loss of 99.999% of all humans, after which enough humans would survive that they could eventually repopulate Earth and perhaps the galaxy.
(emphasis mine)

Luke seems to be saying that even if humanity has a miserable existence for a couple thousand years trying to scramble back to the comforts of civilization, this should give way to millions or billions of years in much better existence. He’s willing to take that deal over extinction, as I think most of us would. None of us are bemoaning the brutish lives of our ancestors, wishing that they had never existed, after all. Rather, we consider ourselves lucky to have a much better existence today, and can be thankful to the countless generations who endured, growing little by little, until we got here.

As for the philosophical stance that anguished existence is better than none – even though, again, I don’t think this has been put forth on this blog – I would think that this is true only if there is a possibility of improving one’s conditions. Like you, I find nonexistence preferable to existence in eternal hell .

  (Quote)

Mark March 20, 2011 at 8:49 pm

Mark,See Chalmers’ paper.  

I take Chalmers’ answer to the question I raised to lie on page 20, where discusses support for what he names the “proportionality thesis.” However, doesn’t the support he enlists seem pretty weak to you? If there are diminishing returns on intelligence amplification for minds with cognitive architectures like ours, we should expect to see large differences at the very beginning of the AI spectrum (e.g., between a primitive computer and nothing) produced by a slight upward tweak of human intelligence (e.g., between Turing and an average person). To disprove the diminishing returns thesis, you have to gauge the size of ascents across multiple stages of the spectrum.

  (Quote)

JS Allen March 20, 2011 at 9:13 pm

Polymeron,

Right; Luke’s position seems to be a leap of faith, ignoring the possibility that the continued existence of humanity might be miserable. It seems that there is some level of sustained human misery beyond which existence of the human race would be undesirable.

If the source of both Armageddon and evil-AI is our technological prowess, it’s not clear that we could “eventually repopulate the galaxy” before we inflicted thousands of more years of misery or extinction upon ourselves. If Luke’s premise is correct, in that we only have 200 more years to address this threat, then it implies that a future civilization would have to live at substantially lower than our standard of living in perpetuity if it hoped to be sustainable. At best, it would experience short periods of giddy hope followed by huge collapses and long climbs back to the point of despair. IMO, the existence of humanity might well be worthwhile, even if we were condemned to a perpetual dark ages, but there are limits to this.

In fact, there is a very real possibility that an unfriendly AI would not make humans extinct, but would instead cause human misery on a scale that we can scarcely imagine. To such a superintelligence, we might appear as no more significant than beasts or even bacteria. Considering how we humans treat beasts, that ought to make us shudder. In the book of Genesis, humans were initially not to slaughter animals, and our treatment of animals is considered to be a consequence of our fallen state. But if Genesis is any guide, if we endow the singularity with a creation myth, it could be 3,000 years before it comes to its senses and considers human rights, in the way that we are just now considering animal rights.

Finally, let’s imagine that 150 years from now, we finally realize that malevolent AI is inevitable unless we bomb humanity back into the stone age. At just the last moment, we succeed in averting an eternity of torment at the hands of a merciless intelligence, and can pass only a short message on to our descendants. What would that message be? That seems to me to be a very important question, since we would want to look for evidence of that message in our ancient past.

  (Quote)

JS Allen March 20, 2011 at 9:43 pm

BTW, did anyone see Bill Maher yesterday? He said, “Republicans are like meth addicts; always focused on imaginary problems”. :-)

I can see how people could compare the singularity alarmists to meth addicts. However, I think Luke’s instincts are right, even if I disagree with his decision calculus. The next 200 years are most likely when the stuff hits the fan, and very few people are taking it seriously enough. If you want to get a historical perspective on the problem, read James Gleick’s fantastic new book, “Information”. History is littered with corpses who underestimated the speed and power of this new paradigm advance — he aptly calls it a “flood” on the book’s cover.

Our ancient ancestors warned us about this, way back in 1984. If you can get past the big hair and tight jeans, Ronny James Dio brings the wisdom.

  (Quote)

Vlastimil Vohánka March 21, 2011 at 2:02 am

Luke,

slightlely offtopic, but related and important: what about substantial technological decline due to declining ERO(E)I and oil depletion.

Very concise summary of the problems, by Dr. Charles Hall (State University of New York), at http://netenergy.theoildrum.com/node/4678. (More fully, at http://www.springerlink.com/content/q31418v301002426 ) I’ve heard Hall believes the problems it probable the technology of the near future will not be higher than the industrial one.

Personally, it seems to me there is some (alarming) evidence both for the peak global oil and the _hitherto_ declining EROI. Yet, I am not certain the extrapolation we won’t be able to solve this (for we haven’t been so far) is justified.

I just would like to know your opinion.

Personally, I incline to the view of another prominent scholar in the field — Dr. Václav Smil (University of Manitoba in Winnipeg, Canada) — that even if we _cannot_ predict exactly or with a firm handle of the probabilities and the threat is serious and substantially new, mankind generally seems to have such good track record and actual resources that it may well handle the situation even without substantial technological decline (cf. his books “Energy: A Beginner’s Guide”, pp. 171-76, “Energy in Nature and Society”, p. 363, and his paper http://www.vaclavsmil.com/wp-content/uploads/docs/smil-article-2006-worldwatch.pdf ).

  (Quote)

Luke Muehlhauser March 21, 2011 at 2:29 am

Alexander Kruel,

For the record, I’m not that confident in my assertion that the best thing to do with your money is spend it on Friendly AI development.

Moreover, I don’t think I’ve given a convincing argument for the claim here, either. I can’t do so in such little space. Mostly, I just wanted to introduce people to the idea of human extinction.

  (Quote)

Polymeron March 21, 2011 at 2:42 am

JS Allen,

I agree with your opening statement:

It seems that there is some level of sustained human misery beyond which existence of the human race would be undesirable.

However, each and every single one of the scenarios you have depicted is exceedingly unlikely. In addition a few are irrelevant.

If the source of both Armageddon and evil-AI is our technological prowess, it’s not clear that we could “eventually repopulate the galaxy” before we inflicted thousands of more years of misery or extinction upon ourselves.

Obviously if we’re going to go extinct then a few extra thousand years of total misery are obviously the lesser option. However there is no reason to assume that our chances of going extinct after another stone age would be any higher than they are now.
Any other outcome – including 10 more stone ages and then transcendence – would be clearly preferable. 10,000 years of suffering for several billion years of good existence? Even given tiny odds, we can just shut up and multiply. They’d need to be ultra-tiny for us to consider being better off dead. And I don’t see why our odds should be considered small, let alone ultra-tiny.

If Luke’s premise is correct, in that we only have 200 more years to address this threat, then it implies that a future civilization would have to live at substantially lower than our standard of living in perpetuity if it hoped to be sustainable. At best, it would experience short periods of giddy hope followed by huge collapses and long climbs back to the point of despair.

This absolutely does not follow from Luke’s premise in any way I can see. As a result, I won’t be analyzing the utility of a perpetual dark ages.

In fact, there is a very real possibility that an unfriendly AI would not make humans extinct, but would instead cause human misery on a scale that we can scarcely imagine.To such a superintelligence, we might appear as no more significant than beasts or even bacteria.Considering how we humans treat beasts, that ought to make us shudder.

That this would lead to a perpetual dark ages full of suffering doesn’t seem very likely, does it? Admittedly, an evil AI overlord is pretty much as likely as a human extinction AI (less likely, because the sheer amount of configurations we would simply not survive). But since it’s essentially the same problem – Friendly AI – it seems like a moot choice between extinction and non-extinction. In either case we need to address the problem.

But if Genesis is any guide, if we endow the singularity with a creation myth, it could be 3,000 years before it comes to its senses and considers human rights, in the way that we are just now considering animal rights.

This fails on multiple levels.
1. There is no good reason to take Genesis as a guide on this. But if we ignore this, then still…
2. There is no good reason to think that a super AI’s timescale for changing opinions would even resemble human civilizations. But even assuming it did…
3. 3,000 or 10,000 years are again insignificant compared to the billions ahead. I guess I’m telling you to “shut up and multiply”.

Finally, let’s imagine that 150 years from now, we finally realize that malevolent AI is inevitable unless we bomb humanity back into the stone age.At just the last moment, we succeed in averting an eternity of torment at the hands of a merciless intelligence, and can pass only a short message on to our descendants.What would that message be?That seems to me to be a very important question, since we would want to look for evidence of that message in our ancient past.  

“Let’s imagine” a genie suddenly appears and will transport you to a desert island, and you could only take one thing with you. What would that be?
That scenario seems about equally likely to what you have described. Your scenario is far too specific to be considered anything more than an interesting thought experiment. No realistic doomsday scenario would play out like this.

  (Quote)

mister k March 21, 2011 at 6:56 am

“Alexander Kruel,

For the record, I’m not that confident in my assertion that the best thing to do with your money is spend it on Friendly AI development.

Moreover, I don’t think I’ve given a convincing argument for the claim here, either. I can’t do so in such little space. Mostly, I just wanted to introduce people to the idea of human extinction.”

I’m glad you’ve admitted to this, because this is a frequent less wrong claim that I disagree with. It effectively pushes one down to a pascals mugging, where imaginary numbers are made up about human lives saved by a super AI, which then means no matter how small I assess the probability of the institude making a difference I should donate most of my money towards them. This ignores the following problems

1-The institute might fail to learn how to make a machine friendly, either because its impossible, because their approach is poor, or they are a fraudulent enterprise
2-Once the institute tells us how to make AI friendly, there seems to be absolutely nothing to compel the makers of AI to listen to them
3-Theres no guarentee that this super computer is necessarily possible. While its definitely true that there exists the potential for greater than human intelligence, its not clear how much greater that would be.
4-Many of the powers we attribute to this super computer rely on its ability to solve problems that may not be solvable. For instance, it may be that there is no way to travel faster than the speed of light which means we are trapped in our system until the sun dies
5-Do future humans have as much value as present humans? There are multiple causes we can support which will definitely help currently existing humans, rather than potentially existing humans.

  (Quote)

Alexander Kruel March 21, 2011 at 8:16 am

@mister k

Looks like someone has been reading some of my comments over at LW. A few corrections:

1-Some of the various problems I mentioned to show that the LW community is concerned with friendly AI but never mentions unfriendly humans, although we have more evidence that such beings are possible.
2-One of many arguments I mentioned in the past to show that it is very unlikely that the MIRI can succeed and that social engineering, public relations and politics play a significant role in trying to mitigate risks from AI.
3-We don’t know enough about the nature of general intelligence to conclude that explosive recursive self-improvement is possible and that any artificially human-level intelligence would run efficiently on various substrates. I think that the noisiness and patchwork architecture of the human brain could play a significant role.
4-The question is if intelligence can be applied to itself effectively. Is intelligence a meta-solution or just an effective searchlight for unknown unknowns? How rare are unknown unknowns and can their discovery be sped up deliberately? That is in my opinion highly questionable. Another problem is that many solutions might be nearly perfect or not subject to exponential improvement either because one hits diminishing returns or the complexity of the problem and amount of data does rise faster and for longer than the improvement of intelligence and computational resources.
5-The problem with imaginary beings is, what if most possible beings are too different from us to peacefully or happily coexist with us? I think it is highly questionable that the extrapolation of human volition could result in an unanimous outcome. Any preference equilibrium would likely be a result of a feedback loop between that which is extrapolating and the subjects of its extrapolation, i.e. artificially induced and the result of its original design parameters rather than what each of the subjects would actually want on their own and without superhuman interference. Here the questions arise, what difference is there to wireheading and what if the only way to maximally stimulate our evolutionary template is to evolve on and under our own control? A whole different problem is the one you mentioned, to what extent should we disregard currently existing beings in favor of potentially existing beings? Here I believe that the long-term future is a problematic factor in calculating the value of mitigating present problems. You might argue that the fate of a galactic civilization is more worthy of our attention than the fate of people who might die due to climate change. But this argument can always be made. Why not disregard a galactic civilization in favor of many more potentially beings that could be alive if we managed to escape the heat death of the universe? The probability of success might be even tinier than the one of giving birth to a galactic civilization but that can be outweighed by the utility of an eternal future. One might argue that we simply don’t know enough about cosmology to reasonably consider such a scenario, but the same argument can be made about the possibility of superhuman AI and colonizing the galaxy. Where do we draw the line? I feel we shouldn’t neglect obvious problems too much and first gather more data and broaden our understanding of some underlying problems before deciding to put all our resources into the outcome of some low probability scenario, regardless of the payoff.

  (Quote)

Curt March 21, 2011 at 8:33 am

Mister K,
You beat me to the punch. I am going to say something along similar lines.
I once tried to devise political institutions that would prevent a small special interest group from ever gaining control over the institutions of the state. I eventually gave up.
No matter how many checks and balances I tried to put in to a system I could always think of a way to overcome those obsticals that would allow a secret organization to take over any government. In fact I came to the conclusion that the more people that there are involved with trying to prevent a domestic conpiracy the more like it will be that a domesitc conspiracy will be successful.
Of course I was working on this project alone in my basement and you know what they say about 2 heads being better than one.
I can not say what the chances are that a machine would become malevolent but I suspect that if it is possible and we humans want to make it theoretically impossible we will find that task impossible. Of course if the machine can think but can not act I would think it rather easy. If such a machine were unleashed by one side of humans in a confict with other humans hopefully the machine can develope a sense of humor.
I wonder what are the chances that we are the product of a machine that destoyed its makers regretted it and then recreated them to watch them stumble around in this dark and cold universe and carry on never ending arguements abouts Gods and socialism vs. capitalism and about pacifism vs. class war or anti imperialist violence.
We can not say of course. The Shadow knows.
Concrete Condemnation Curt

  (Quote)

Bill Williams March 21, 2011 at 9:20 am

@Luke

Bill,What you’ve suggested is the opposite of what I recommended. I think we should give to charity to fund the development of non-killing machines. Not killing machines.  

The two are inseparable. If you’re overcoming technical hurdles to the development of friendly superintelligent machines, you’re also advancing the development of unfriendly superintelligent machines, e.g., the U.S. military has a serious hard-on for this shit and will eagerly adopt new technologies that advance their global ambitions. Another evolutionary arms race is quite possible — this time with intelligence rather than plutonium.

How about writing a blog post that discusses the morality of a small group of enthusiasts independently deciding to accelerate the development of a technology that many concede could destroy 7 billion lives? Remember — the beliefs of many experts necessarily lead to this possibility. Those who deny this point are basing their unsupported optimism on subjective speculation.

Even if the risk of opening Pandora’s Box is only 1%, is it rational for a small group of enthusiasts to make that decision for everyone else. Shouldn’t we all get to decide? At what percent of risk is it reasonable for the few to decide for the many?

I realize that superintelligent machines may be coming either way. I’m trying to understand why good-hearted people are so eager to accelerate this process.

  (Quote)

Alexander Kruel March 21, 2011 at 9:37 am

The two are inseparable. If you’re overcoming technical hurdles to the development of friendly superintelligent machines, you’re also advancing the development of unfriendly superintelligent machines…

Working on friendly AI does not necessarily mean to work on artificial general intelligence directly but rather mathematical formalisms of human values, goal-stability and general decision theoretical problems. Some of the people working on friendly AI are basically mathematical philosophers who try to formalize a sort of foolproof ethical framework that someone working on artificial general intelligence could then implement into the goal-system of an AGI to guarantee its safety under recursive self-improvement.

  (Quote)

Bill Williams March 21, 2011 at 10:01 am

@Alexander Kruel said,

Some of the people working on friendly AI are basically mathematical philosophers who try to formalize a sort of foolproof ethical framework …

Are you saying that such efforts do not in any way advance the development of technologies that could be used for harm? I find that hard to believe.

Assholes will eagerly tweak new discoveries and insights to advance their malicious agenda.

  (Quote)

Polymeron March 21, 2011 at 10:13 am

Bill Williams,

Somehow I don’t see the weaponized mathematical formulation of charity being deployed on the battlefield any time soon. Friendly AI is a research field that appears, at a glance, to be purely catastrophe-preventing.
Also by your logic, ANY new discoveries and insights are too dangerous to be worth researching.

As for the morality of advancing Friendly AI, advocates would tell you that it is the best way to prevent unfriendly AI from being created. Consider that a lot of people have access to your Pandora’s box; they want to open it first so as to have the best chances of avoiding a catastrophe, before someone less thorough does it first.

You can still argue that they are wrong about this, but frankly I don’t see a practical way of limiting AI research and it also seems impractical to wait for humanity to reach consensus on this matter. So I’m not sure what alternative you are proposing.

  (Quote)

JS Allen March 21, 2011 at 10:16 am

However, each and every single one of the scenarios you have depicted is exceedingly unlikely.

Obviously. We’re talking about a Pascal’s mugging, after all. The question is whether my scenarios are more or less likely than the scenario of a friendly superintelligent AI, since that’s what Luke wants to guarantee. IMO, a superintelligence that inadvertently causes immense human misery is quite a bit more likely than a superintelligence that cares about our “ethics”.

However there is no reason to assume that our chances of going extinct after another stone age would be any higher than they are now.

Actually, there is good reason to assume that. The scenario posited in the original post presumes that humanity would choose the stone age as an alternative to extinction. If humanity were at the brink of extinction due to superintelligence, such that we found the stone age to be a preferable outcome, there is no reason to assume that we could ever return to a high standard of living without incurring the same existential risk.

In other words, we need to posit a plausible outcome where we would have interstellar travel, without having the kind of computation infrastructure which would risk a malevolent singularity. I don’t see how we can do that. Are we going to have interstellar travel without AI?

Admittedly, an evil AI overlord is pretty much as likely as a human extinction AI (less likely, because the sheer amount of configurations we would simply not survive). But since it’s essentially the same problem – Friendly AI – it seems like a moot choice between extinction and non-extinction. In either case we need to address the problem.

I agree that an inadvertently malicious AI is far more likely than either a specifically friendly or malevolent AI. But the idea that we would “simply not survive” seems like wishful thinking. We are already at the point where puny humans can clone one another; mass farming of humans for reasons inscrutable to us would be child’s play for a superintelligent AI.

You’re welcome to say that an AI that inadvertently perpetuates eternal human misery is “essentially the same problem” as an AI that inadvertently wipes us out. Presumably, then, you would agree with Alexander Pruss in saying that existence in hell is better than non-existence? I don’t know exactly where I stand on that issue, but it’s at least as real a possibility as the possibilities outlined in the original post.

This fails on multiple levels

Agreed, for the reasons you cite. I was just expressing extreme skepticism at the idea that a superintelligence would treat us any better than we treat beasts — even if Luke were to succeed in programming it with a creation myth designed to cause it to be “ethical”. Do you have any reason to be more hopeful?

Your scenario is far too specific to be considered anything more than an interesting thought experiment. No realistic doomsday scenario would play out like this.

Agreed. The most likely outcomes are either that we’re doomed, or we’re not. The outcome where we’re almost doomed, and pull back at the last minute, is far less probable. However, it’s the scenario that we would most likely be able to find evidence for, if it ever happened in the past.

  (Quote)

Polymeron March 21, 2011 at 10:36 am

JS Allen,

Obviously.We’re talking about a Pascal’s mugging, after all.
The question is whether my scenarios are more or less likely than the scenario of a friendly superintelligent AI, since that’s what Luke wants to guarantee.IMO, a superintelligence that inadvertently causes immense human misery is quite a bit more likely than a superintelligence that cares about our “ethics”.

I think that we simply don’t have the information necessary to answer this question – yet another reason to research Friendly AI.

Actually, there is good reason to assume that.The scenario posited in the original post presumes that humanity would choose the stone age as an alternative to extinction. If humanity were at the brink of extinction due to superintelligence, such that we found the stone age to be a preferable outcome, there is no reason to assume that we could ever return to a high standard of living without incurring the same existential risk.In other words, we need to posit a plausible outcome where we would have interstellar travel, without having the kind of computation infrastructure which would risk a malevolent singularity.I don’t see how we can do that.Are we going to have interstellar travel without AI?

You are positing a huge amount of low probability assumptions here, including that demolishing all our knowledge is the only safety. This scenario is absurdly unlikely, and is likely functionally impossible.
“But it has immense negative utility”, you say. Well tough break, I do not accept Pascal’s mugging as a valid argument. If you do, then you should totally send me all your money right now, because I totally have a button that kicks 4^^^4 puppies and I’m not afraid to use it if you do not comply with my demands ;)

You’re welcome to say that an AI that inadvertently perpetuates eternal human misery is “essentially the same problem” as an AI that inadvertently wipes us out.Presumably, then, you would agree with Alexander Pruss in saying that existence in hell is better than non-existence?

This absolutely does not follow, at least in any way I can see. And I already said I disagree with that statement.
You are making leaps of logic that I simply don’t understand. You’ll need to better explain your reasoning if you want me to address it.

I was just expressing extreme skepticism at the idea that a superintelligence would treat us any better than we treat beasts — even if Luke were to succeed in programming it with a creation myth designed to cause it to be “ethical”.Do you have any reason to be more hopeful?

I have reason to research into how such a state of affairs may be brought about. Until we have, we can’t correctly ascertain its likelihood.

I don’t think we have any profound disagreement on this. I think the main point is that, in dealing with such vast utilities, the best thing you can do is improve your margin of confidence.

  (Quote)

JS Allen March 21, 2011 at 10:54 am

I wonder what are the chances that we are the product of a machine that destoyed its makers regretted it and then recreated them to watch them stumble around in this dark and cold universe and carry on never ending arguements abouts Gods and socialism vs. capitalism and about pacifism vs. class war or anti imperialist violence.

David Chalmers (the zombie guy) thinks there is a 20% chance that we are living in a simulation. This possibility, even if very small, has bearing on Luke’s question.

Just as we use the supercollider to attempt to map out the laws of physics in “string space”, perhaps some trans-dimensional physicist has created us as a means of mapping out the laws of physics for a corner of the universe he can’t inhabit. If that were the case, we should drag our feet a bit and sandbag our science, since he’s bound to shut off the simulation when he gets the answers he’s looking for. Why rush things? And in this case, I wouldn’t want to be the guy who is talking about consuming the simulator’s resources to create AI; friendly or otherwise — the creator of the sim would have already put in place safeguards to prevent misuse of simulator resources.

I don’t think we have any profound disagreement on this. I think the main point is that, in dealing with such vast utilities, the best thing you can do is improve your margin of confidence

Yes, that’s precisely the point. Luke’s original post was a bit like saying “There might be a dragon in that cave that will destroy the entire village; we should all dedicate our lives to figuring out how to train dragons to be nice!” Others are simply pointing out that it might be smarter to get the hell away from the cave, or seal it with concrete.

When there is an opportunity cost for investing resources, pouring money into “research on friendly AI” is not a way of “improving your margin of confidence”. It’s a commitment that presumes a level of confidence that is unjustified. If you want to improve your margin of confidence, that’s probably the last place you would invest money.

  (Quote)

Bill Williams March 21, 2011 at 11:00 am

@Polymeron said,

I don’t see the weaponized mathematical formulation of charity being deployed on the battlefield any time soon.

If we figure out how to make machines behave ethically, assholes can tweak those discoveries to impose their own warped sense of ethics. How could it be any other way?

Also by your logic, ANY new discoveries and insights are too dangerous to be worth researching.

You’re putting words in my mouth – that’s not what I said. Any rational person would agree that we need to carefully consider the implications before developing new technologies, and this particular technology is scary as hell. I’m deeply concerned about the “go, go, go” mentality of AI enthusiasts. I think we need more careful deliberation and less unbridled enthusiasm.

As for the morality of advancing Friendly AI, advocates would tell you that it is the best way to prevent unfriendly AI from being created.

That’s just more unsupported subjective speculation.

I don’t see a practical way of limiting AI research and it also seems impractical to wait for humanity to reach consensus on this matter. So I’m not sure what alternative you are proposing.

I’m questioning why we’re blindly accelerating the process. I understand that there is no practical way to vote, but I’d love to hear a coherent moral explanation of how a small group of enthusiasts can blindly justify putting the entire species at risk, especially considering that no one has a crystal ball.

  (Quote)

Alexander Kruel March 21, 2011 at 11:13 am

@Everyone

“There might be a dragon in that cave that will destroy the entire village; we should all dedicate our lives to figuring out how to train dragons to be nice!”

My problem with this is that people seem to be asking “What is the expected utility of training dragons to be nice?” rather than “What evidence do you have that there might be a dragon in that cave?”. I don’t have a satisfactory answer to why the first question is problematic, I can only cite Pascal’s Mugging and ask, what’s the difference? I don’t think that calculating the expected utility of an outcome is generally wrong but that once one enters the realm of weighing vast amounts of utility against astronomical low probabilities one might be well advised to ask about empirical evidence.

If I am simply confused about this, please enlighten me or or link me up.

  (Quote)

Curt March 21, 2011 at 11:55 am

Do AI enthusiasts communticate with each other on an international level frequently?
I do not mean on an institutional level but on a personal level. I ask this because it is a variation of the two heads are better than one rule. It seems plausible to me that if people are working on this in national groups and not as international groups they will be less effective, for better or worse, because just as individuals have blind spots so to can a group all with a similar national background. Furthermore if an international group has more disagreements to work through it will slow things down, for better or worse.
c.c.c. can anyone find me a P.

  (Quote)

Luke Muehlhauser March 21, 2011 at 12:52 pm

Curt,

There are thousands of AI researchers in the world, and they publish in journals. But there’s too much being published for anybody to keep up with even 1/100th of it.

  (Quote)

Polymeron March 21, 2011 at 1:14 pm

JS Allen,

When there is an opportunity cost for investing resources, pouring money into “research on friendly AI” is not a way of “improving your margin of confidence”.It’s a commitment that presumes a level of confidence that is unjustified.If you want to improve your margin of confidence, that’s probably the last place you would invest money.  

That is plausible. However you should factor in the following:
1. Inherent risks of getting to Friendly AI too late. I don’t think the probability for that is insignificant.
2. The obvious utility in (not to mention the inevitability of) building intelligent machines. Even if Singularity scenarios are not possible, knowledge gained researching Friendly AI could still find applications in designing safer intelligent machines.
3. It also increases our understanding of how complex self-feeding loops behave after many iterations, which could be useful for more applications.

I also reject your claim that this will not improve our margins of confidence. We might not get from FAI research an answer to whether or not the MIRI is possible, but if we found that FAI is impossible, this would definitely be a critical piece of evidence to weigh later on. I do agree that other research can better tell us if the danger is real, but it seems to me that the meager resources going to this field make sense in light of all of the above. They do not seem wasted.

Bill Williams,

If we figure out how to make machines behave ethically, assholes can tweak those discoveries to impose their own warped sense of ethics. How could it be any other way?

Formulated morality will not just tell you how to better make an AI converge on your a specific set of moral principles. Rather, it would tell you if your principles are actually moral. All your “asshole” would get from this research would be a confirmation that he or she is, in fact, an asshole.

You’re putting words in my mouth – that’s not what I said. Any rational person would agree that we need to carefully consider the implications before developing new technologies, and this particular technology is scary as hell.

Do I think AI is scary? Yes. How about Friendliness research? Not s0 much.

I still think I see a bit of a double standard here.
A lot of things have scary implications if taken to an imaginary extreme. I don’t see you calling for people to stop genetics research in case someone weaponizes that to make a GeneSelect weapon that kills everyone predisposed to mental disorders or obesity. Nor are we opposed to genetic modification research – even the anti-GM-crop crowd just wants to block the application, rather than the research.
Now, if you want to say that research aimed at producing an AI should not be accelerated due to risks, that is a reasonable position. However categorically bunching into that research fields aimed at mitigating said risk seems less than reasonable. Please reconsider this position…

I’m deeply concerned about the “go, go, go” mentality of AI enthusiasts. I think we need more careful deliberation and less unbridled enthusiasm.That’s just more unsupported subjective speculation.
I’m questioning why we’re blindly accelerating the process. I understand that there is no practical way to vote, but I’d love to hear a coherent moral explanation of how a small group of enthusiasts can blindly justify putting the entire species at risk, especially considering that no one has a crystal ball.  

This has not clarified what alternative you are suggesting. I see no reason to discuss the merits or pitfalls of their approach when no alternative has been set forth.

Also see my next post, which I have split off from this one for brevity and clarity, but may further frame the discussion regarding the risks and justifications. If you can bear with its length, that is.

  (Quote)

Polymeron March 21, 2011 at 1:14 pm

Alexander Kruel,

@Everyone
My problem with this is that people seem to be asking “What is the expected utility of training dragons to be nice?” rather than “What evidence do you have that there might be a dragon in that cave?”. I don’t have a satisfactory answer to why the first question is problematic, I can only cite Pascal’s Mugging and ask, what’s the difference? I don’t think that calculating the expected utility of an outcome is generally wrong but that once one enters the realm of weighing vast amounts of utility against astronomical low probabilities one might be well advised to ask about empirical evidence.If I am simply confused about this, please enlighten me or or link me up.  

One serving of enlightenment, coming up. Warning: This is a bit long, but bear with me.

Pascal’s Mugging deals with a situation where the evidence for something being true is, for all purposes, nonexistent. For it to be true, everything you know must be wrong. (e.g. “I will use my powers from beyond the Matrix to kill 3^^^3 people).

In contrast, this situation relies on the following assumptions:
1) Motivated intelligence can be artificially constructed. This is highly probable, because our current information shows that intelligence is the product of natural processes occurring in a physical system (i.e. brains).
2) Motivated intelligence can be made with the tools currently at our disposal. This again is highly likely: Computers are very close to the universal Turing machine, and we already have enough computing power to simulate neuronal components equal to a human brain (and beyond). Personally I find the Neural Network approach inefficient, brute-force-like and generally distasteful, but the point remains that this premise is highly likely.
3) AI is dangerous. This, at first glance, might seem like the really contentious point, but again I don’t see how it could be wrong – intelligence is dangerous, period. Even if the AI never designs anything (much less its predecessor) and is never given a lot of resources, it would still pose a hazard, much like any person does if they are not a stable member of society.

This should be sufficient to show that some safety research is warranted. I hope we can at least agree on this much before we continue! However, it does not touch on the vast utilities you say are unwarranted. So let’s examine the next few premises.

4) General Artificial Intelligence could be taught to design other AI. Again, this is exceedingly likely. At very least we could mimic human intelligence if we understood it well enough, and humans can design AIs.
5) Humans would use existing AIs at least as help for designing other AIs. This might be a bit contentious, since exactly the appearance of risk might stop us from doing so by regulatory means. However, the wide availability of the resources needed to make AI makes it unlikely that such regulation would be sufficient, and in addition the line between the general AI and non-general AI is fuzzy: We are already using less clever algorithms as assistance on optimization problems and the like. So this premise remains very likely.
6) When work is delegated by intelligent agents to other intelligent agents, infidelities to the original requirements tend to crop up. Now, if you’ve ever played Chinese Whispers, I don’t need to explain to you why this is true. And note some of the loss of fidelity could be to safety requirements, even if the result is inspected by humans.

So far, I think I’ve shown that Friendly AI research is justified even in the absence of self-improving machines or super-intelligent AIs. It’s just a good idea to have machines around that can design other machines, IF we can make the results safe.

But we’re still in the realm of high probabilities and mundane utilities… Let’s delve in further, to murkier waters.

7) Artificial Intelligence far greater than human is possible. I’d rate this as highly likely, because with the AI technology already available you could at very least have a lot different intelligences talking to each other, which would mimic the collective human intelligence. So at very least we are talking about the possibility of a humanity-equivalent intelligence. The only objection I could see to this is if it would take immense resources to run even one human-equivalent intelligence, and this could never be improved on. This scenario seems to me exceedingly unlikely considering our current technology, not to mention future advancement.
8) Humans would allow AIs to get far more intelligent than humans through successive improvement. This is the most contentious claim, I think. In the naive scenario, AIs accomplish so much for us that we become dependent on them. If we are too wary of the risk to go down that path, scenarios such as the AI in the box being smart enough to trick us sound convincing (though they should by no means be assigned a very high probability). Regulation could be strong, and there could well be an upper limit to how strong a nonregulated AI could get before getting shut down. But, is this premise really unlikely? You have humans building bombs in their basement, so if a person manages to build a nonregulated AI, is it far fetched to think that a shoddily built AI would nevertheless be clever enough to hide its existence until it was very powerful?

Even if you consider all the scenarios on this to be unlikely, none of them are astronomically unlikely, certainly not when put together. They do not require everything you know to be wrong. They don’t even necessitate going as far as the preposterous utilities of the paperclip solar system or the galaxy-inhabiting AI; it is enough to consider a simple corruption of an election’s results or the systematic mapping of all known diseases within our lifetime. They are not Pascal’s Mugger, just an ordinary mugger with a gun that may or may not be loaded. And when you are facing vast utilities hinging on only somewhat unlikely outcomes, that is when you shut up and multiply.

If I have made an error along the way, I would be happy to accept corrections.

P.S. I’m aware I’ve been using undefined probability values, but if you accept premises 1-7 to be at least 95% probable each, and would agree that premise 8 is at least 10% probable given the first seven, then you have a very meaningful chance on your hands that is nothing like Pascal’s Mugger. I am of course open to any of the premises being contested as falling short of these rule of thumb probabilities, but please phrase objections accordingly.

  (Quote)

Curt March 21, 2011 at 1:22 pm

Wow Facsinating. I had no idea. And to think, I thought for a few momemts, if people are working on this in secret small groups it could have potenially destabilizing results at some point in the future. Why for example a high tech computer team from a militarist faction in the US could try to design AI that could overcome the protections that another country has put in place to make sure that the election results of its computer ballots are actually an accurate reflection of the votes cast.
Maybe I should be a bit more specific. Say for example that this faction wanted to launch an attack on Iran. Well if it did that it would be pretty easy to figure that one consequence of this action would be a huge leftward shift in the electorate of Germany.
This faction would certiantly want to avoid that so they may contract a team of specialists to to worm their way inside the German voting systems and use primative AI to do it. That way they could reassure nervous nellies in their ranks that Germany will still be a good place to do business even after the US launched such an attack.
Why such a group may even have a back up plan to sabotage the German Chancellors helicopter while she is on a campaign tour and after it crashes blame it on Iranian agents, or Al Quida. Or maybe if the Chancellor found out about this plan and stopped it she suddenly fell out of grace with this faction. It all seems just like a espionge thriller that I just made up off the top of my head. It is and it was not very hard.
I have not seen this reported anywhere outside of Germany but last Wednesday the Helicopter the the Chancellor was riding in, which just entered service in December of 2010 plunged straight down for thousands of feet after power went out in both engines. The pilot managed to get the engines restarted just several hundred feet off the ground and seconds before impact to avoid a crash. The incident is of course under investigation. Sabotage has not been ruled out but it is of course not expected.
Happy New Years by the way.
Condescending Confetti Curt

  (Quote)

Polymeron March 21, 2011 at 1:54 pm

Curt,

There is no practical, near-term purpose that can be achieved faster with a General AI than with currently existing computing methods.

Even if General AI was invented tomorrow and the “recipe” for how to write one was published to the Internet, it would take years before the General AI was better than specialized systems at pretty much any task you can think of.

While it’s possible that certain design ideas could inspire a better narrowly functional system, this is no more dangerous than any other form of widely available academic knowledge.

  (Quote)

Bill Williams March 21, 2011 at 2:23 pm

@Polymeron

You make some good points. If you’re proposing that the advancement of friendly AI is purely a philosophical exercise about how to be more moral, and won’t, for example, generate mathematical constructs that could be tweaked to create unfriendly AI, I’m all for it. It’s not clear to me that friendly AI enthusiasts are generally willing to accept that limitation.

I don’t think it’s fair to compare my objections with the implications of genetic research. I’m not just dreaming up wild scary scenarios. Leading AI experts say that it’s quite possible, or even probable, that we’ll lose control of superintelligent entities. That scary scenario speaks for itself.

There is a reason that we’re on top of the food chain – because we’re the smartest critters on the block. It’s not clear to me why we would be so enthusiastic about creating something that is smarter than us, especially since we don’t know if we can control it, how it will evolve, how it will be used, how it will behave, etc. etc.

I see no reason to discuss the merits or pitfalls of their approach when no alternative has been set forth.

I don’t have an alternative approach. I’m saying don’t accelerate the development of superintelligence until we determine that it’s in our best interest to do so. Currently no one knows, which should concern everyone.

The claims of the top experts in this field taken to their logical conclusion suggests that we might be creating the ultimate doomsday machine. That sobering reality compels me to prefer caution over enthusiasm.

  (Quote)

JS Allen March 21, 2011 at 2:27 pm

Alexander,

I don’t think you’re confused. At this point in history, there are still very few people who take the idea of a dragon in the cave seriously. Personally, I think the odds are fairly high — maybe 20% probability in the next 200 years. But intelligent people can differ.

Polymeron already touched on some of the reasons that I think the odds are high. I’ll add a bit more:

We don’t know enough about the nature of general intelligence to conclude that explosive recursive self-improvement is possible and that any artificially human-level intelligence would run efficiently on various substrates. I think that the noisiness and patchwork architecture of the human brain could play a significant role.

I think we know enough to be legitimately concerned. The MIRI needn’t do everything we consider to be “general intelligence” to be a threat — it need only be vastly superior to us in types of intelligence that pose an existential threat to us. We can enumerate some of those areas, and we know that some are already areas of active investment, and that the computation power scales pretty effectively.

We may discover that some forms of intelligence meet rapidly diminishing returns in scale/power. But I think it’s highly unlikely that this will affect many of the forms of intelligence that pose a threat.

The question is if intelligence can be applied to itself effectively. Is intelligence a meta-solution or just an effective searchlight for unknown unknowns? How rare are unknown unknowns and can their discovery be sped up deliberately? That is in my opinion highly questionable.

Again, it depends on what type of intelligence we’re talking about. We already have systems that make themselves smarter via “genetic algorithms”, so that’s one obvious mechanism that an intelligent system could seek to become more intelligent — basically through guided trial and error against a certain environment. Another way that systems become more intelligent is through increasing the size of their training data. It is possible today to build a computer than can process every word that has ever been written by humans, in a matter of hours.

  (Quote)

mister k March 21, 2011 at 3:43 pm

@ Alexander Kruel I have seen a few of your comments (I’m thakil at less wrong), but most of those thoughts were my own.

@Polymeron. Your arguments are reasonable, but don’t really address the concerns that I raised. In particular: 8 seems not necessarily true to me, and even if it is we have to assume that the institute will actually help. What bothers me about this tower of possibilities that, if true, the bad guys could just make bad AI, and by bad guys I mean any millitary force. Now while its true that the millitary have managed to avoid firing nukes, we know that they came VERY close to doing so, even with a palpable demonstration of their destructive power. A convincing demonstration of the destructive power of AI is only likely to occur once it exists.

I’m not at all opposed to research into friendly AI, I think it does good, and it may end up helping a lot, I’m just not convinced its necessary to save existence.

  (Quote)

Polymeron March 22, 2011 at 12:25 am

mister k,

I was not claiming that premise 8 is necessarily true; note my final note that says it just needs to have a 10% chance of being true in order for the utility involved to be vast. And specifically it works with your own premises 1-4 (which essentially constitute my premise 8). Nor did I claim that Friendly AI research is necessary to save existence.

My argument was simply that it is a good idea, and in fact I made the point that it is a good idea even if premises 7-8 are false.

Now, if you’re saying Luke hasn’t shown that the best thing to do with your money is donate it to the institute, then I agree (and he said he does, too). However I am convinced that Friendly AI research is a worthy cause, and I’ve shown why.

We don’t seem to have any disagreement I can discern.

Bill Williams,
Sans any regulatory steps to try and actively inhibit AI research, I don’t see how we can be said to be “speeding it up”. Some individual people have dedicated themselves to research AI, that is true. You can say that those people are not justified in doing so. However I don’t think picketing their offices or anything of the kind would help change their mind and make them abandon the field, so again this remains purely academic.
I think that in a practical sense, we need to accept that people are already researching AI, for whatever reasons; and think what we want to do with that knowledge.

  (Quote)

cl March 22, 2011 at 2:56 am

Bill Williams,

I’m deeply concerned about the “go, go, go” mentality of AI enthusiasts. I think we need more careful deliberation and less unbridled enthusiasm.

Uh… yeah!

  (Quote)

Curt March 22, 2011 at 3:54 am

@JS Allen,
I am going to come out of the closet now. What I am about to tell you I have told only one other person. In order to maintain my low level of standing in the economic communty I have kept this to my self for 3 or 4 years now.
I believe that this whole universe is a simultion, probably for research and or entertainment purposes. I will not at all attmept to defend my belief because I can not offer a shred of evidence that would be accepted as scienitific. I can offer only a few loose threads that could at best be described as bad comedy.
More importantly knowing that we are a simuation would change nothing in my opinion. The whole point of ethics IMO is to help us decide on both on an individual and collective level what should we be doing that we are not doing now and what should we not be doing now that we are doing. I think the fact that we have a centeral nervous system still sets one key but on some occasions overrideable principle to our behavior which is to seek pleasure and avoid pain including psychological pain.
It also seems apparent to me that our human evolution has given most of us, but not all, humans some sense that values some type of fairness although our environment plays a huge factor in determining what exactly we think of as fair most of us do not take pleasure in killing or harming others of our species that we do not deem a threat to us or our values. This acts as a constraint on the drive to seek pleasure and avoid pain preventing most of us from acting like psychopaths.
Now just for a couple of things that I view as loose threads in our history.
There are a bunch of key words in Buddhism, which may also occur in Hinduism a relgion that I have never studied, that would today make wonderful accronyms for the US military which loves to employ acronyms. My favorite is Samsara which would be an acronym for Same Attitude Means Same Action Repeated Again which really explains the Buddhists meaning of Samsara and the cycle that the US military is caught in. While we are here discussing possible threats to humans exsistance in 200 years we have an institution here today that fullfills that role.
Then there is the word God itself which could easily be an accronym for General Organizational Department. as opposed to the word doG Department Organization General which really annoys the english speaking members of the GOD centeral committee who sought to descredit the word amongst the human population but only succeeded in the Muslim part. You may say, ” Hey wait a minute. The Arabic word for dog is not dog. ” Yes they are clever arent they. They do everything in a way that leaves plausible denyabilty if they get caught.
Then there is the biblical story of the loyalty test that God gave to Abraham. If we think this through we can come to no other correct conclusion than that God is actually Satan with different hat. God put Abraham in a position that no matter what Abraham did God could have praised him or condemned him. Of course for thousands of years people have been reaching different conclusions about this loyalty test but they have been comming to the wrong conclusion becasue I do not approve of their conclusions.
If you do not agree then all that I can say is that you will have to figure it out for yourself.
There are other senseless reasons that I believe that this universe is a simulation but I will not seriously attempt to convert people because if this simulation really is for research purpopes if to many people reached the same conclusion that I have it might seriously skew the results.
After all when humans thak part in simulations designed by psychologists the volunteers taking part are always told exactly and truthfully what the ultimate purpose of the test is so that they can act in a manner that gives the test results that would please them.

  (Quote)

Curt March 22, 2011 at 4:00 am

Yes in a narrow sense I contridicted myself with my closing but in a broad sense I think that it is still consitent. Hypocrasy is a kind of contradiction and I think that it is very difficult to make it through this life, unless it is very short, without at times being a hypocrite.

  (Quote)

Alexander Kruel March 22, 2011 at 4:04 am

@Polymeron

If I have made an error along the way, I would be happy to accept corrections.

No error, I actually donated 3 times to the Machine Intelligence Research Institute in the past. What I am objecting is that all those conclusions are not evidence based enough to make friendly AI a priority right now, neglecting other risks and problems. The scope of the commitment within the friendly AI community seems unjustified. There are many questions, 1) will the development of artificial general intelligence happen quickly or slow enough to adapt 2) does human-level intelligence run efficiently on various substrates or is it also a hardware problem 3) can intelligence be applied to itself effectively enough as to allow explosive recursive self-improvement etc. Even if you answer all possible objections with Yes, what does that mean regarding friendly AI charity? Currently the MIRI does receive $500k per year, isn’t that sustainable given the current projects and state of knowledge? If they told me what exactly they would do with more money I might very well agree that a charitable contribution to friendly AI research could have the largest impact.

They do not require everything you know to be wrong.

The same line of reasoning could be employed for other charities as well, for example space elevator research supporting space colonization. One could list a lot of things that could go wrong on Earth and that we could only survive if we colonized space. Not all of the assumptions have to be right to make it reasonable to leave Earth. Yet the most important question is still if space colonization is feasible right now, does it make sense to devote a lot of resources to it at this time? The same could be said about friendly AI, do we really know enough to started working on friendly AI or should we first gather more evidence about the nature of intelligence? It is probably smart to research space colonization, even if we’ll only be able to colonize the solar system, but to what extent? You better don’t neglect other obvious problems that might very well wipe you out as well.

  (Quote)

Polymeron March 22, 2011 at 4:35 am

Alexander Kruel,

I agree with pretty much everything you said. I think the conclusion is we could try to calculate the probabilities and multiply by the utility – assuming we actually had a utility meter. That could give us a good idea as to whether AI research, space elevator research, or anything else is on par with other causes, or possibly different from them by several orders of magnitude (in either direction).

At this point the probabilities seem high enough that we are justified in investing some resources into it. I haven’t yet seen anything that would convince me that we should drop everything else right now and divert all efforts to it; so on that point we certainly agree.

Looks like something of a consensus is starting to brew :)

  (Quote)

Bill Williams March 22, 2011 at 8:50 am

mister k said,

Sans any regulatory steps to try and actively inhibit AI research, I don’t see how we can be said to be “speeding it up”.

Um, this might be a good time to review the blog post that initiated this discussion. Here’s the part that got under my skin:

…the single largest impact you can have with your charity dollars is to give all of them toward ensuring we develop artificial superintelligence that is friendly to human goals. Giving to stop global warming looks like a drop in a puddle in comparison.

  (Quote)

mister k March 22, 2011 at 10:27 am

“mister k said,

Sans any regulatory steps to try and actively inhibit AI research, I don’t see how we can be said to be “speeding it up”.”

Um… I’m not sure I did.

  (Quote)

cl March 22, 2011 at 11:40 am

Bill Williams,

I keep returning to the automobile in my mind, along with almost any other technological innovation. Like you, I’m skeptical of the “go, go, go” mentality. That mentality is responsible for who-knows-what-percentage of our current environmental / natural resource problems. It’s only a matter of time before we dig ourselves a hole we can’t climb out of, if we haven’t already. Personally, I find it ridiculous that Luke appears to be so naïve here. Coming from somebody who ostensibly worships science and claims to eschew intuition on account of all the times it’s been wrong before, I find the claim that “we [should] develop artificial superintelligence that is friendly to human goals” particularly suspect. Where’s the empirical evidence? If none, why all the “anti-intuition” rhetoric?

Anyways, don’t mind me. I’m just trying to make sense where none seems forthcoming, and your comments inspired a little bit of that.

  (Quote)

Bill Williams March 22, 2011 at 1:12 pm

My bad mister k — sorry. I was responding to Polymeron.

  (Quote)

Bill Williams March 22, 2011 at 1:26 pm

Thanks for the comment CL. To Luke’s credit, he clarified his position in his comment above on March 20, 2011 at 1:35 pm:

I am indeed in favor of slowing progress in artificial general intelligence.

That’s an admirable example of what Eliezer Yudkowsky would call updating your map to the evidence. In light of what science has discovered about neuroplasticity, belief perseverance, and other cognitive influences, that is a remarkably quick adjustment.

Kudos to Luke!

  (Quote)

cl March 22, 2011 at 1:48 pm

Bill Williams,

In contrast to those who only spout negativity and derision, I have routinely demonstrated my ability to say “Kudos to Luke!” on this blog. Of course, I’m also more than willing to say “Luke, bad!” when and where I think it’s appropriate, and my point was that his claim I cited seems out-of-line with his stated penchant for empirical evidence, and his stated distrust of intuition. IOW:

I am indeed in favor of slowing progress in artificial general intelligence.

Kudos to Luke!

we [should] develop artificial superintelligence that is friendly to human goals

Luke bad!

  (Quote)

cl March 22, 2011 at 1:51 pm

Oh, and one other thing:

That’s an admirable example of what Eliezer Yudkowsky would call updating your map to the evidence.

Perhaps, but I’m not nearly as in awe of Eliezer Yudkowsky as Luke and/or others, so this doesn’t mean much to me.

  (Quote)

Bill Williams March 22, 2011 at 2:21 pm

@cl,

Perhaps, but I’m not nearly as in awe of Eliezer Yudkowsky as Luke and/or others, so this doesn’t mean much to me.

You don’t need to admire the guy to appreciate his wisdom. He’s a seriously smart cookie with an impressive capacity for rational thought.

If we assume that my views about AI are more rational than Eliezer’s, then he is seriously misguided on that particular issue — that’s called being human.

To avoid committing the logical fallacy of poisoning the well, I’m compelled to consider his arguments, and I’m glad I do. He has impressed me many times.

  (Quote)

cl March 22, 2011 at 3:47 pm

Bill,

To avoid committing the logical fallacy of poisoning the well, I’m compelled to consider his arguments…

As am I. I suppose he’s just impressed me a few less times than he’s impressed you. At any rate, my primary concern in posting to this thread was to call into question Luke’s claim. I see a discrepancy between his stated penchant for empirical evidence and the claim he made, which seems to rely on the very intuition he calls into question. In short, I’m confused.

  (Quote)

Luke Muehlhauser March 22, 2011 at 11:26 pm

cl,

Yeah, it’s not about Eliezer, it’s about updating beliefs in response to evidence. Do you disagree with that, somehow?

  (Quote)

cl March 23, 2011 at 10:09 pm

Luke,

Yeah, it’s not about Eliezer…

I didn’t say it was. Bill brought Eliezer up, not me.

…it’s about updating beliefs in response to evidence. Do you disagree with that, somehow?

Would I say “kudos to Luke” in response to your doing that which I disagree with? Of course not, and I have a hard time believing you didn’t read the above comment where I said:

I see a discrepancy between [Luke's] stated penchant for empirical evidence and the claim he made, which seems to rely on the very intuition he calls into question.

That’s what this is about.

  (Quote)

Bill Williams March 24, 2011 at 8:00 am

@cl

Statement 1: Perhaps, but I’m not nearly as in awe of Eliezer Yudkowsky as Luke and/or others, so this doesn’t mean much to me.

Statement 2: Bill brought Eliezer up, not me.

It doesn’t matter who first used Eliezer’s name. This is a lame-ass attempt to distract readers from the issue. Luke is correctly noting an inconsistency within your statements (further supported by your history of behavior on this blog).

Statement 3: As am I. (CL’s response to the comment: “To avoid committing the logical fallacy of poisoning the well, I’m compelled to consider his [Eliezer's] arguments…”)

Statements 1 & 3 directly contradict each other, i.e., you’re talking out of both sides of your mouth.

You jumped in at the tail end of a long conversation and have been confrontational and irrational, i.e., logical fallacies flow freely from your trash talk.

You and I have tangled before. Your last words to me were a promise that you would never talk to me again. Once again, that would be awesome!

  (Quote)

cl March 24, 2011 at 12:06 pm

Great. Now I suspect Luke will simply nod his head saying “Yup!” to Bill Williams, instead of actually responding to the apparent discrepancy between his stated position on intuition and his speciesist claim that seems entirely based on intuition. Way to go, Bill!

Bill Williams,

I agreed with Luke that this is not about Eliezer, and you were the one who brought him up. Those are undeniable, incontrovertible facts, whether you like them or not.

Statements 1 & 3 directly contradict each other, i.e., you’re talking out of both sides of your mouth.

False. Stating my lack of concern for what Eliezer Yudkowsky would call updating one’s map to the evidence does not entail poisoning the well, whatsoever. A contradiction is an instance of X and ~X, not a wild card you can pull whenever you want to rail against someone who thinks a little lower of somebody you’re obviously quite enamored with.

You jumped in at the tail end of a long conversation and have been confrontational and irrational, i.e., logical fallacies flow freely from your trash talk.

False, and the bit about “trash talk” seems quite ironic a sentence or two after,

This is a lame-ass attempt to distract readers from the issue.

Get real. It’s obvious you’ve got yourself all upset over nothing. If you wish to add something salient to my confusion over Luke’s stated penchant for eschewing intuition and Luke’s claim that seems entirely based on intuition, feel free. Else, on to the next.

  (Quote)

cl March 24, 2011 at 12:07 pm

Oh, one other thing:

Your last words to me were a promise that you would never talk to me again.

I honestly don’t recall that, but you might be right. Since you’re so rational, and I’m so irrational, may I politely request the evidence for this claim?

  (Quote)

Sonny October 29, 2011 at 11:34 am

When you all speak of A.I. What makes you think the A.I needs to be brought into physical realm being able to move with it’s own limbs? We only need it’s intelligence to create an more simpler ‘automata’ to be able to control nanotech amongst anything else. There will be ways of restricting A.I movements.

Realistically we should be worried about engineered viruses.

  (Quote)

Polymeron October 29, 2011 at 4:02 pm

Sonny, “needing its intelligence” is a whole can of worms in that regard.

Yes, you can get an added layer of safety by ensuring that the AI only has a simple interface (e.g. text-based interface), and any machine or consequence of its intelligence is actually done by human hands. But that still leaves 2 distinct possibilities:

1. The AI hides something in the code it provides to us – since it’s super-intelligent, it could be something we’d miss (if we were smart enough to build it ourselves, we would have…). Such a thing could be a program to extract the AI into a more powerful apparatus.

2. The AI could manipulate humans via the text interface into giving it more power. If it understands human psychology well enough, and has the necessary resources, it’s not impossible that it could mind-control a human being via text interface, and get the human to “let it out of the box”, as it were.

Or not even mind control. Imagine, for instance, that you give the AI control over a simple sound device so it can interact in speech. Sounds harmless enough, right? But if it is capable of emitting a sound in a volume or frequency that can cause you intense pain, it could force you to do anything while you’re physically in the room – say, give it Internet access. Or what if it finds a way to run its electronic circuitry in a way that transmits electronic pulses (data) to remote devices, even if you didn’t actually equip it with a wireless card? This is exactly the sort of things we’re liable to miss when designing precautions, if we’re not careful.

I agree that keeping the AI boxed gives you some measure of defense; but once it’s out, it’s out. And I think that when discussing super-intelligent beings, we should not be restricted by a lack of imagination.

  (Quote)

Leave a Comment

{ 2 trackbacks }