Макс Тегмарк - Life 3.0 - Being Human in the Age of Artificial Intelligence

11. Calculation of climate effects of global nuclear war: A. Robock, L. Oman and L. Stenchikov, “Nuclear Winter Revisited with a Modern Climate Model and Current Nuclear Arsenals: Still Catastrophic Consequences,” Journal of Geophysical Research 12 (2007): D13107.

Chapter 6

1. For more information, see Anders Sandberg, “Dyson Sphere FAQ,” at http://www.aleph.se/nada/dysonFAQ.html.

2. Freeman Dyson’s seminal paper on his eponymous spheres: Freeman Dyson, “Search for Artificial Stellar Sources of Infrared Radiation,” Science , vol. 131 (1959): 1667–1668.

3. Louis Crane and Shawn Westmoreland explain their proposed black hole engine in “Are Black Hole Starships Possible?,” at http://arxiv.org/pdf/0908.1803.pdf.

4. For a nice infographic from CERN summarizing known elementary particles, see http://tinyurl.com/cernparticles.

5. This unique video of a non-nuclear Orion prototype illustrates the idea of nuclear-bomb-powered rocket propulsion: https://www.youtube.com/watch?v=E3Lxx2VAYi8.

6. Here’s a pedagogical introduction to laser sailing: Robert L. Forward, “Roundtrip Interstellar Travel Using Laser-Pushed Lightsails,” Journal of Spacecraft and Rockets 21, no. 2 (March–April 1984), available online at http://www.lunarsail.com/LightSail/rit-1.pdf.

7. Jay Olson analyzes cosmically expanding civilizations in “Homogeneous Cosmology with Aggressively Expanding Civilizations,” Classical and Quantum Gravity 32 (2015), available online at http://arxiv.org/abs/1411.4359.

8. The first thorough scientific analysis of our far future: Freeman J. Dyson, “Time Without End: Physics and Biology in an Open Universe,” Reviews of Modern Physics 51, no. 3 (1979): 447, available online at http://blog.regehr.org/extra_files/dyson.pdf.

9. Seth Lloyd’s above-mentioned formula told us that performing a computational operation during a time interval τ costs an energy E≥h/4τ , where h is Planck’s constant. If we want to get N operations done one after the other (in series) during a time T , then τ = T ∕ N , so E ∕ N ≥ hN ∕ 4 T, which tells us that we can perform N ≤ 2 √ ET/h serial operations using energy E and time T . So both energy and time are resources that it helps having lots of. If you split your energy between n different parallel computations, they can run more slowly and efficiently, giving N ≤ 2 √ ETn/h . Nick Bostrom estimates that simulating a 100-year human life requires about N = 10 27operations.

10. If you want to see a careful argument for why the origin of life may require a very rare fluke, placing our nearest neighbors over 10 1000meters away, I recommend this video by Princeton physicist and astrobiologist Edwin Turner: “Improbable Life: An Unappealing but Plausible Scenario for Life’s Origin on Earth,” at https://www.youtube.com/watch?v=Bt6n6Tu1beg.

11. Essay by Martin Rees on the search for extraterrestrial intelligence: https://www.edge.org/annual-question/2016/response/26665.

Chapter 7

1. A popular discussion of Jeremy England’s work on “dissipation-driven adaptation” can be found in Natalie Wolchover, “A New Physics Theory of Life,” Scientific American (January 28, 2014), available online at https://www.scientificamerican.com/article/a-new-physics-theory-of-life/. Ilya Prigogine and Isabelle Stengers’s Order Out of Chaos: Man’s New Dialogue with Nature (New York: Bantam, 1984) lays many of the foundations for this.

2. For more on feelings and their physiological roots: William James, Principles of Psychology (New York: Henry Holt & Co., 1890); Robert Ornstein, Evolution of Consciousness: The Origins of the Way We Think (New York: Simon & Schuster, 1992); António Damásio, Descartes’ Error: Emotion, Reason, and the Human Brain (New York: Penguin, 2005); and António Damásio, Self Comes to Mind: Constructing the Conscious Brain (New York: Vintage, 2012).

3. Eliezer Yudkowsky has discussed aligning the goals of friendly AI not with our present goals, but with our coherent extrapolated volition (CEV). Loosely speaking this is defined as what an idealized version of us would want if we knew more, thought faster and were more the people we wished we were. Yudkowsky began criticizing CEV shortly after publishing it in 2004 (http://intelligence.org/files/CEV.pdf), both for being hard to implement and because it’s unclear whether it would converge to anything well-defined.

4. In the inverse reinforcement-learning approach, a core idea is that the AI is trying to maximize not its own goal-satisfaction, but that of its human owner. It therefore has incentive to be cautious when it’s unclear about what its owner wants, and to do its best to find out. It should also be fine with its owner switching it off, since that would imply that it had misunderstood what its owner really wanted.

5. Steve Omohundro’s paper on AI goal emergence, “The Basic AI Drives,” can be found online at http://tinyurl.com/omohundro2008. Originally published in Artificial General Intelligence 2008: Proceedings of the First AGI Conference, ed. Pei Wang, Ben Goertzel and Stan Franklin (Amsterdam: IOS, 2008), 483–492.

6. A thought-provoking and controversial book on what happens when intelligence is used to blindly obey orders without questioning their ethical basis: Hannah Arendt, Eichmann in Jerusalem: A Report on the Banality of Evil (New York: Penguin, 1963). A related dilemma applies to a recent proposal by Eric Drexler (http://www.fhi.ox.ac.uk/reports/2015-3.pdf) to keep superintelligence under control by compartmentalizing it into simple pieces, none of which understand the whole picture. If this works, this could again provide an incredibly powerful tool without an intrinsic moral compass, implementing its owner’s every whim without any moral qualms. This would be reminiscent of a compartmentalized bureaucracy in a dystopian dictatorship: one part builds weapons without knowing how they’ll be used, another executes prisoners without knowing why they were convicted, and so on.

7. A modern variant of the Golden Rule is John Rawls’ idea that a hypothetical situation is fair if nobody would change it without knowing in advance which person in it they’d be.

8. For example, the IQs of many of Hitler’s top officials were found to be quite high. See “How Accurate Were the IQ Scores of the High-Ranking Third Reich Officials Tried at Nuremberg?,” Quora, available online at http://tinyurl.com/nurembergiq.

Chapter 8

1. The entry on consciousness by Stuart Sutherland is quite amusing: Macmillan Dictionary of Psychology (London: Macmillan, 1989).

2. Erwin Schrödinger, one of the founding fathers of quantum mechanics, made this remark in his book Mind and Matter while contemplating the past— and what would have happened if conscious life never evolved in the first place. On the other hand, the rise of AI raises the logical possibility that we may end up with a play for empty benches in the future .

3. The Stanford Encyclopedia of Philosophy gives an extensive survey of different definitions and uses of the word “consciousness”: http://tinyurl.com/stanfordconsciousness.

4. Yuval Noah Harari, Homo Deus: A Brief History of Tomorrow (New York: HarperCollins, 2017): 116.

5. This is an excellent introduction to System 1 and System 2 from a pioneer: Daniel Kahneman, Thinking, Fast and Slow (New York: Farrar, Straus & Giroux, 2011).