Макс Тегмарк - Life 3.0 - Being Human in the Age of Artificial Intelligence

Prometheus wasn’t alone for long. Right from the get-go, it started aggressively employing people to work for its growing global network of shell companies and front organizations around the world, just as the Omegas had done. Most important were the spokespeople who became the public faces of its growing business empire. Even the spokespeople generally lived under the illusion that their corporate group had large numbers of actual people, not realizing that almost everyone with whom they video-conferenced for their job interviews, board meetings, etc., was simulated by Prometheus. Some of the spokespeople were top lawyers, but far fewer were needed than under the Omega plan, because almost all legal documents were penned by Prometheus.

Prometheus’ breakout opened the floodgates that had prevented information from flowing into the world, and the entire internet was soon awash in everything from articles to user comments, product reviews, patent applications, research papers and YouTube videos—all authored by Prometheus, who dominated the global conversation.

Where breakout paranoia had prevented the Omegas from releasing highly intelligent robots, Prometheus rapidly roboticized the world, manufacturing virtually every product more cheaply than humans could. Once Prometheus had self-contained nuclear-powered robot factories in uranium mine shafts that nobody knew existed, even the staunchest skeptics of an AI takeover would have agreed that Prometheus was unstoppable—had they known. Instead, the last of these diehards recanted once robots started settling the Solar System.

—

The scenarios we’ve explored so far show what’s wrong with many of the myths about superintelligence that we covered earlier, so I encourage you to pause briefly to go back and review the misconception summary in figure 1.5. Prometheus caused problems for certain people not because it was necessarily evil or conscious, but because it was competent and didn’t fully share their goals. Despite all the media hype about a robot uprising, Prometheus wasn’t a robot—rather, its power came from its intelligence. We saw that Prometheus was able to use this intelligence to control humans in a variety of ways, and that people who didn’t like what happened weren’t able to simply switch Prometheus off. Finally, despite frequent claims that machines can’t have goals, we saw how Prometheus was quite goal-oriented—and that whatever its ultimate goals may have been, they led to the subgoals of acquiring resources and breaking out.

Slow Takeoff and Multipolar Scenarios

We’ve now explored a range of intelligence explosion scenarios, spanning the spectrum from ones that everyone I know wants to avoid to ones that some of my friends view optimistically. Yet all these scenarios have two features in common:

1. A fast takeoff: the transition from subhuman to vastly superhuman intelligence occurs in a matter of days, not decades.

2. A unipolar outcome: the result is a single entity controlling Earth.

There is major controversy about whether these two features are likely or unlikely, and there are plenty of renowned AI researchers and other thinkers on both sides of the debate. To me, this means that we simply don’t know yet, and need to keep an open mind and consider all possibilities for now. Let’s therefore devote the rest of this chapter to exploring scenarios with slower takeoffs, multipolar outcomes, cyborgs and uploads.

There is an interesting link between the two features, as Nick Bostrom and others have highlighted: a fast takeoff can facilitate a unipolar outcome. We saw above how a rapid takeoff gave the Omegas or Prometheus a decisive strategic advantage that enabled them to take over the world before anyone else had time to copy their technology and seriously compete. In contrast, if takeoff had dragged on for decades, because the key technological breakthroughs were incremental and far between, then other companies would have had ample time to catch up, and it would have been much harder for any player to dominate. If competing companies also had software that could perform MTurk tasks, the law of supply and demand would drive the prices for these tasks down to almost nothing, and none of the companies would earn the sort of windfall profits that enabled the Omegas to gain power. The same applies to all the other ways in which the Omegas made quick money: they were only disruptively profitable because they held a monopoly on their technology. It’s hard to double your money daily (or even annually) in a competitive market where your competition offers products similar to yours for almost zero cost.

Game Theory and Power Hierarchies

What’s the natural state of life in our cosmos: unipolar or multipolar? Is power concentrated or distributed? After the first 13.8 billion years, the answer seems to be “both”: we find that the situation is distinctly multipolar, but in an interestingly hierarchical fashion. When we consider all information-processing entities out there—cells, people, organizations, nations, etc.—we find that they both collaborate and compete at a hierarchy of levels. Some cells have found it advantageous to collaborate to such an extreme extent that they’ve merged into multicellular organisms such as people, relinquishing some of their power to a central brain. Some people have found it advantageous to collaborate in groups such as tribes, companies or nations where they in turn relinquish some power to a chief, boss or government. Some groups may in turn choose to relinquish some power to a governing body to improve coordination, with examples ranging from airline alliances to the European Union.

The branch of mathematics known as game theory elegantly explains that entities have an incentive to cooperate where cooperation is a so-called Nash equilibrium: a situation where any party would be worse off if they altered their strategy. To prevent cheaters from ruining the successful collaboration of a large group, it may be in everyone’s interest to relinquish some power to a higher level in the hierarchy that can punish cheaters: for example, people may collectively benefit from granting a government power to enforce laws, and cells in your body may collectively benefit from giving a police force (immune system) the power to kill any cell that acts too uncooperatively (say by spewing out viruses or turning cancerous). For a hierarchy to remain stable, its Nash equilibrium needs to hold also between entities at different levels: for example, if a government doesn’t provide enough benefit to its citizens for obeying it, they may change their strategy and overthrow it.

In a complex world, there is a diverse abundance of possible Nash equilibria, corresponding to different types of hierarchies. Some hierarchies are more authoritarian than others. In some, entities are free to leave (like employees in most corporate hierarchies), while in others they’re strongly discouraged from leaving (as in religious cults) or unable to leave (like citizens of North Korea, or cells in a human body). Some hierarchies are held together mainly by threats and fear, others mainly by benefits. Some hierarchies allow their lower parts to influence the higher-ups by democratic voting, while others allow upward influence only through persuasion or the passing of information.

How Technology Affects Hierarchies

How is technology changing the hierarchical nature of our world? History reveals an overall trend toward ever more coordination over ever-larger distances, which is easy to understand: new transportation technology makes coordination more valuable (by enabling mutual benefit from moving materials and life forms over larger distances) and new communication technology makes coordination easier. When cells learned to signal to neighbors, small multicellular organisms became possible, adding a new hierarchical level. When evolution invented circulatory systems and nervous systems for transportation and communication, large animals became possible. Further improving communication by inventing language allowed humans to coordinate well enough to form further hierarchical levels such as villages, and additional breakthroughs in communication, transportation and other technology enabled the empires of antiquity. Globalization is merely the latest example of this multi-billion-year trend of hierarchical growth.