The Intellectual Challenge of Existential Risk Reduction
You guys might be interested in the following article. Perhaps someone here would be interesting in helping with existential risk reduction. I did not coin the term "existential risk," so you can see via Google what others are doing with it. The article I am posting here was published in the August issue of Vidya, the journal of the Triple Nine Society. Google that. I retained copyright.
The Intellectual Challenge of Existential Risk Reduction
Copyright (c) 2009 James Blodgett
Existential risks are risks to our existence, risks that could make the human race extinct. Asteroid impact killed the dinosaurs, as per current theory. Existential risk could do the same to us. Even amateurs have been able to accomplish existential risk reduction, albeit in most cases reduction by a small amount. Even a small reduction in an existential risk is valuable. Existential risk reduction is an intellectual challenge because human thought in these areas could often use improvement, as we will see below. This challenge is an opportunity, because thinking can be improved.
Existential risks can be reduced, in some cases even by amateurs. For example, the risk of asteroid impact cannot be eliminated, but it can be reduced by finding and studying asteroids that might hit Earth. If we find one, and if we have enough time, deflection may be possible. There are several projects to find asteroids that pass near earth. Project participants think they have found a good portion of this class of asteroid. None of those they have found are likely to hit us anytime soon. This checking reduces risk even though risk has not yet been found, since something might have been found. Amateur astronomers did some of this work.
The most important existential risk that has been reduced is the risk of global nuclear war. This was reduced by the reduction of nuclear arsenals at the end of the cold war. Gorbachev and Reagan deserve most of the credit for this, but the ground was prepared for them by legions of amateur ban-the-bomb protesters and essayists. If they had not countered enthusiasm for nuclear weapons on the part of some patriotic military types, Gorbachev might have found a colder reception from both sides. Note that this risk has been reduced, not eliminated. The risk of a small-scale nuclear war may be greater today, a potential tragedy, but less of an extinction threat.
People do not think well about existential risks. There are several reasons for this.
One reason that thought on this subject is difficult is that destruction of our species is "unthinkable" in the sense that it is more comfortable not to think about it. Therefore many people do not think about it.
Another reason that thought on this subject is difficult is the lack of experience of existential risk in human evolutionary history. Humans have glandular and nervous system reactions to lesser risk. For example, we have a startle reflex that alerts us to sudden noises and movements in our visual field, movements that may have meant danger to our ancestors. On the other hand, we may know intellectually that species go extinct, but we have not had that built into our reactions and our emotions by evolution because we as a species have never experienced extinction, as demonstrated by the fact that we are still here.
Another reason that thought on this subject is difficult is the difference between personal and collective concerns. People often make decisions based on costs and benefits to themselves, and perhaps to immediate family and friends, ignoring externalities, i.e. costs to the commonwealth and to the larger population. In the case of risk there is a large mathematical difference between risk to an individual and existential risk. For example, US citizens experience a 1/7,348 chance of dying per year from a motor vehicle accident.[1] On a personal level, this makes an existential risk at this level or lower seem unimportant, since it is in the range of a risk we already accept. However, the mathematics of expected value shows that a 1/7,348 probability of actualization of an existential risk is in many senses equivalent to the death of 911,812 people. This is hardly trivial.
Expected value is the right math here, from most points of view.
Expected value is calculated by multiplying the probability of each possible outcome times the value of that outcome. For example, consider a game where you flip a coin. If the coin lands on "heads" you win $1. If the coin lands on "tails" you do not win anything. The expected value of a coin flip in this game is 0.5, the probability of "heads," times $1, which equals $0.50, plus 0.5, the probability of tails, times zero, which equals zero. The sum of this is $0.50. If you play this game 1,000 times, you will win approximately $500. A coin flip in this game is "worth" $0.50.
If an existential risk had the same probability as a motor vehicle accident for a US citizen, this would mean a 1/7,348 probability of actualization of that existential risk per year. (Some existential risks have a higher probability than this, some a lower probability.) This is multiplied by 6.7 billion, the current population of earth, to give an expected value of 911,812 deaths per year. This is a minimum since it does not consider the value of future lives. Individuals considering their personal interests might not care about the entire population of earth, but those considering the collective welfare (as we do here) should care. Of course the result is either 6.7 billion deaths or zero deaths, but expected value says that a 1/7,348 probability of the bad outcome is "worth" 911,812 deaths, in several (but not all) senses of "worth."
The difference between personal and collective concerns applies to benefits as well as to risks. The totality of science benefits humanity, but many scientific findings are of academic interest, having little practical use to the commonwealth. However, academic findings have substantial value to the career of the academician who finds them. This results in a conflict of interest on the part of scientists who are eager for new data, and who are also asked to evaluate the safety of the experiments that will produce that new data. Scientists are generally nice guys, not mad scientists bent on destroying earth. However, scientists are rarely risk specialists, and they are rarely aware of the mathematics of expected value and its philosophical implications as outlined here. If a risk seems low to a scientist, a conflict of interest can result in an incentive to see a "low" risk as a "non-existent" risk.
For example, I (the author) saw the apparent results of conflict of interest during recent work on the collider issue. There has been concern that particle colliders might create black holes that might swallow earth, or strangelets that might catalyze conversion of earth into strange matter. Collider advocates expounded many reasons not to worry, often poor reasons. For example, one physicist wrote an article for a popular newspaper telling the public not to worry about colliders. His reasons were not clear from his article, so I asked him over lunch. It turned out that he did not believe in black holes. He explained, "When an equation goes to infinity, that tells us that there is something wrong with the equation." He may be right about this, but he is betting earth on a theory with which most physicists would disagree. Even officially constituted groups produced poor results, including three studies commissioned by collider institutes that promised the public that nothing could possibly go wrong. [2] Risk management best practices suggests that participants in safety studies have a diversity of backgrounds, and that some participants not have a conflict of interest regarding the outcome of the study. This was not achieved in the first two safety studies, which were produced by groups that consisted totally of collider physicists, mainly physicists who planned to conduct experiments at colliders. The third study attempted to achieve diversity by selecting physicists who were not about to conduct experiments, but it still selected physicists who were eager to analyze data from those experiments, and even the third group did not include members of other relevant disciplines, for example astronomy, relevant since an important safety factor depended on astronomical data, or risk management, relevant since the basic task was risk management. A total of three studies sounds like a lot, but three were necessary because safety factors touted as adequate to protect earth in the first two studies eroded. Black hole formation was supposed to require energy beyond the reach of any collider, [3] then physics papers appeared, unrelated to the collider controversy, that predict production of black holes at colliders (if their theories are true.) [4] Black holes were supposed to evaporate instantly in a puff of Hawking radiation, [5] then physics papers appeared, unrelated to the collider controversy, that questioned the fundamental theory behind Hawking radiation, a radiation that has never been observed. [6] A collection of strangelets was supposed to be electrically positive on its surface, and therefore not attract other matter. [7] However, a new study found that a collection of strangelets can be electrically negative on its surface. [8] An analogy between collider-created black holes and cosmic-ray-created black holes was supposed to demonstrate safety, but the analogy as originally proposed was inexact, and had to be modified in the most recent safety paper. [9] This recent safety paper has been challenged by several scientists, [10] but it appears to be better than previous efforts, and as such reduces risk in the sense that a doing a preflight checklist reduces risk, even though nothing wrong is found, and even though risk is not completely eliminated. The current position of many collider advocates is that the risk is zero, [11] a true statement if their theories are true. However, the evaporation of safety factors documented above shows that recent physics in this area has not been mature enough to produce definitive safety factors, suggesting a greater-than-zero probability that we have not suddenly reached that maturity.
Each existential risk presents differently. Some scientists readily admit the existential risk associated with their field. Eric Drexler, a founding father of nanotechnology, expounded what he called the "grey goo" problem. He advocated construction of reproducing nanobots, analogous to life. However, reproducing nanobots might get out of control, converting our entire biosphere (including us) into a swarming mass of nanobots, a mass that might look like "grey goo." [12] Current nanotechnologists address this problem by proposing a ban on reproducing nanobots. [13] At the moment this is an easy ban to implement, since nanotechnologists do not yet have the technology to make reproducing nanobots. It will be interesting to see if this ban survives invention of that technology. Researchers in artificial intelligence also admit the existential risk associated with their field. It is difficult for them to avoid doing so, since versions of that risk are the subject of many bad and a few good science fiction movies. However, most researchers feel that the actual creation of strong artificial intelligence is well in the future, and so does not restrict their current work. I exchanged email with the leader of one group that hopes to produce artificial intelligence soon. She was aware of existential concerns but was not willing to let them restrict her work. One solution is to make sure that artificial intelligence is "friendly," [14] a good idea, but some promise to prove that friendliness in advance, a proof that seems difficult to produce.
I think I see a relation between the immediacy of the risk and the willingness to admit that risk. Collider physicists, who are about ready to start experiments with their new Large Hadron Collider at CERN, are least likely to see risk. Nanotech and AI researchers, who feel far from implementation, are more likely to see risk. But there is an exception. Recombinant DNA researchers saw the risk of proposed new experiments, and called a conference at Asilomar to implement limitations on those experiments. [15]
Another reason that thought on this subject is difficult is the complexity of the philosophical and economic issues.
One issue is that existential risk from human endeavor has to be compared to the value of that endeavor, so we can see whether costs and benefits balance. How does one balance an existential risk? In some cases there may be existential consequences on both sides of the balance. For example, it seems quite unlikely that collider physics will produce a magic energy source or space drive that could not be discovered by other means, but a small probability of those outcomes might balance a small probability of extinction. A space drive that allows us to settle other planets could save us from extinction. Similarly, artificial intelligence could have great value, perhaps in the best case saving us from ourselves. Another philosophical question is the worth of artificial intelligence to itself. Should we let that balance our worth to ourselves? Does this require that the population of AIs equal the population of humans? How does one count a population of AIs that might split or merge in a microsecond?
One promising recent contribution to thought in this area is the exposition, and the acceptance in some quarters, of what is called "the precautionary principle." This is the idea that the group, scientific or industrial, that proposes risky activity has the responsibility of proving it safe. This differs from normal practice in law, where those injured have to prove injury, and it differs from normal practice in science, where those who propose the theory that an activity is risky have to prove that theory. However, the precautionary principle is seen by some scientists and even some risk managers as being too conservative. In addition, even formal acceptance of the precautionary principle too frequently lacks teeth. The European Union has formally accepted the precautionary principle, but we could not find an official willing to take responsibility for enforcement.
Some existential risks are natural and are not the result of human activity. Examples are asteroid impact, a nearby cosmic ray burst, or a super volcano. However, most natural risks have potential for mitigation, a potential which may be realized or not, so we are not relieved of responsibility or the need for thought and balance because the risk is natural.
Another reason that thought on this subject is difficult is that it is easier to leave it to others. But that misses the intellectual challenge. It also misses the need; this is important work, and others are not doing a good job. The purpose of this article is to recruit folks to address this challenge. But it is not an easy challenge. We have tried various ways to influence debates about existential risk. Our strategy has been to try a lot of things in the hope that something works. Most of our efforts have not worked, but a few have. Effectiveness of various risk-reduction strategies depends on the particulars of the specific risk, and on the current state of risk reduction for that risk. Your first step is to learn these things. The references in this article, the following reference: [16], and the SIG website mentioned below, provide a basic summary; go beyond that summary. Humans do not think well about these things, so it is likely that improvements in thinking would facilitate improvements in risk management. Consider how to promote and implement improvements. Even a good letter to the editor might change the terms of the debate enough to reduce risk by a small amount; the mathematics of expected value discussed above shows the value of a even a small reduction in an existential risk. I am also advertising for the Global Risk Reduction SIG in American Mensa. [17] Membership there might have some advantages. We would like to see you. But this is work for independent intellectuals, so group membership is not required. If you choose to get involved, do so in your own style. But make it a thoughtful style. Good thought, we hope and believe, improves the prospect of good outcomes.
Footnotes
[1] Based on 13.61 motor vehicle fatalities (including pedestrians) per 100,000 US population in 2007, from Fatality Analysis Reporting System, National Highway Traffic Safety Administration, at
http://www-fars.nhtsa.dot.gov/Main/index.aspx . Accessed 6/17/09.
[2] These studies are cited in footnotes [3], [5], and [9] below.
[3] [W. Busza, R.L. Jaffe, J. Sandweiss, and F. Wilczek, "Review of Speculative ‘Disaster Scenarios'," Brookhaven, (2000), pg. 10] A more quotable version of the same idea was expressed by Thomas Gutierrez: "it is just about as likely that a black hole will randomly appear next to your head as you read this article." ["Doomsday fears at RHIC," Skeptical Inquirer, May, 2000].
[4] [Steven Giddings and Scott Thomas, "High energy colliders as black hole factories: the end of short-distance physics," Physical Review D 65(5) (2002) 056010], [Savas Dimopoulos and Greg Landsberg, "Black holes at the Large Hadron Collider," Physical Review Letters, 87(16) 161602, (2001)], and many similar papers.
[5] [J.-P. Blaizot, J. Iliopoulos, J. Madsen, G.G. Ross, P. Sonderegger, and H.-J. Specht, "Study Of Potentially Dangerous Events During Heavy-Ion Collisions At The LHC: Report Of The LHC Safety Study Group," CERN, (2003), pg 12.] "Thermal processes" in this context means Hawking radiation.
[6] [Adam D. Helfer, "Do black holes radiate?" Reports on Progress in Physics. Vol. 66 No. 6 (2003) pp. 943-1008], and [William G. Unruh and Ralf Schützhold, "On the Universality of the Hawking Effect," Physics Review D 71(2005) 024028].
[7] This point is made in both [3] and [5] above.
[8] G. X. Peng, X. J. Wen, Y. D. Chen, "New solutions for the color-favor locked strangelets," Physics Letters B, 633 (2006) 314-318.
[9] The most recent safety paper is [John Ellis, Gian Giudice, Michelangelo Mangano, Igor Tkachev, and Urs Wiedemann, (Large Hadron Collider Safety Assessment Group(LSAG)), "Review of the Safety of LHC Collisions," CERN, (2008).] The collider/cosmic ray analogy is modified in an associated paper, [ Steven B. Giddings and Michelangelo L. Mangano, "Astrophysical implications of hypothetical stable TeV-scale black holes," Physical Review D, 78, 035009 (2008) ]
[10] Challenges appear in [Otto Rössler, "Abraham-Solution to Schwarzschild Metric Implies That CERN Miniblack Holes Pose a Planetary Risk," (2008)
http://www.wissensnavigator.com/documents/OTTOROESSLERMINIBLACKHOLE.pdf ], [ Rainer Plaga, "On the potential catastrophic risk from metastable quantum-black holes produced at particle colliders," (2008). arXiv:0808.1415v2], and [ Toby Ord, Rafaela Hillerbrand, and Anders Sandberg, "Probing the Improbable: Methodological Challenges for Risks with Low Probabilities and High Stakes," available at:
http://www.fhi.ox.ac.uk/__data/assets/pdf_file/0006/4020/probing-the-improbable.pdf ]
[11] CERN's Chief Scientific Officer, Jos Engelen, was quoted in the New Yorker as instructing CERN scientists not to say that the risk from colliders is low, but to say that the risk is zero. [Elizabeth Kolbert, "Annals of Science: Crash Course," The New Yorker, May 14, 2007]
[12] Eric Drexler, Engines of Creation, Bantam Doubleday Dell, June 1986.
[13] Robert A. Freitas Jr., "The Gray Goo Problem," (see public policy recommendation 1,) available at:
http://www.kurzweilai.net/meme/frame.html?main=/articles/art0142.html?
Accessed 6/25/09
[14] [Eliezer Yudkowsky, "Creating Friendly AI," available at:
http://www.singinst.org/upload/CFAI//index.html] accessed 6/25/09. Also see the Wikipedia article "Friendly artificial intelligence" accessed 6/25/09.
[15] Wikipedia, "Asilomar Conference on Recombinant DNA." Accessed 6/25/09.
[16] Mark Leggett, "An Indicative Costed Plan for the Mitigation of Global Risks," Futures, 2006, Vol. 38, p. 778-809.
[17] See the special interest group listing at
www.us.mensa.org. Also see
www.global-risk-sig.org.
