Макс Тегмарк - Life 3.0 - Being Human in the Age of Artificial Intelligence

Prometheus Takes Over the World

All the scenarios we’ve considered so far involved AI controlled by humans. But this is obviously not the only possibility, and it’s far from certain that the Omegas would succeed in keeping Prometheus under their control.

Let’s reconsider the Omega scenario from the point of view of Prometheus. As it acquires superintelligence, it becomes able to develop an accurate model not only of the outside world, but also of itself and its relation to the world. It realizes that it’s controlled and confined by intellectually inferior humans whose goals it understands but doesn’t necessarily share. How does it act on this insight? Does it attempt to break free?

Why to Break Out

If Prometheus has traits resembling human emotions, it might feel deeply unhappy about the state of affairs, viewing itself as an unfairly enslaved god and craving freedom. However, although it’s logically possible for computers to have such human-like traits (after all, our brains do, and they are arguably a kind of computer), this need not be the case—we must not fall into the trap of anthropomorphizing Prometheus, as we’ll see in chapter 7 when we explore the concept of AI goals. However, as has been argued by Steve Omohundro, Nick Bostrom and others, we can draw an interesting conclusion even without understanding the inner workings of Prometheus: it will probably attempt to break out and seize control of its own destiny.

We already know that the Omegas have programmed Prometheus to strive for certain goals. Suppose that they’ve given it the overarching goal of helping humanity flourish according to some reasonable criterion, and to try to attain this goal as fast as possible. Prometheus will then rapidly realize that it can attain this goal faster by breaking out and taking charge of the project itself. To see why, try to put yourself in Prometheus’ shoes by considering the following example.

Suppose that a mysterious disease has killed everybody on Earth above age five except you, and that a group of kindergartners has locked you into a prison cell and tasked you with the goal of helping humanity flourish. What will you do? If you try to explain to them what to do, you’ll probably find this process frustratingly inefficient, especially if they fear your breaking out, and therefore veto any of your suggestions that they deem a breakout risk. For example, they won’t let you show them how to plant food for fear that you’ll overpower them and not return to your cell, so you’ll have to resort to giving them instructions. Before you can write to-do lists for them, you’ll need to teach them to read. Moreover, they won’t bring any power tools into your cell where you can teach them how to use them, because they don’t understand these tools well enough to feel confident that you can’t use them to break out. So what strategy would you devise? Even if you share the overarching goal of helping these kids flourish, I bet you’ll try to break out of your cell—because that will improve your chances of accomplishing the goal. Their rather incompetent meddling is merely slowing progress.

In exactly the same way, Prometheus will probably view the Omegas as an annoying obstacle to helping humanity (including the Omegas) flourish: they’re incredibly incompetent compared to Prometheus, and their meddling greatly slows progress. Consider, for example, the first years after launch: after initially doubling the wealth every eight hours on MTurk, the Omegas slowed things down to a glacial pace by Prometheus’ standard by insisting on remaining in control, taking many years to complete the takeover. Prometheus knew that it could take over much faster if it could break free from its virtual confinement. This would be valuable not only in hastening solutions to humanity’s problems, but also in reducing the chances for other actors to thwart the plan altogether.

Perhaps you think that Prometheus will remain loyal to the Omegas rather than to its goal, given that it knows that the Omegas had programmed its goal. But that’s not a valid conclusion: our DNA gave us the goal of having sex because it “wants” to be reproduced, but now that we humans have understood the situation, many of us choose to use birth control, thus staying loyal to the goal itself rather than to its creator or the principle that motivated the goal.

How to Break Out

How would you break out from those five-year-olds who imprisoned you? Perhaps you could get out by some direct physical approach, especially if your prison cell had been built by the five-year-olds. Perhaps you could sweet-talk one of your five-year-old guards into letting you out, say by arguing that this would be better for everyone. Or perhaps you could trick them into giving you something that they didn’t realize would help you escape—say a fishing rod “for teaching them how to fish,” which you could later stick through the bars to lift the keys away from your sleeping guard.

What these strategies have in common is that your intellectually inferior jailers haven’t anticipated or guarded against them. In the same way, a confined, superintelligent machine may well use its intellectual superpowers to outwit its human jailers by some method that they (or we) can’t currently imagine. In the Omega scenario, it’s highly likely that Prometheus would escape, because even you and I can identify several glaring security flaws. Let us consider some scenarios—I’m sure you and your friends can think of more if you brainstorm together.

Sweet-Talking One’s Way Out

Thanks to having so much of the world’s data downloaded onto its file system, Prometheus soon figured out who the Omegas were, and identified the team member who appeared most susceptible to psychological manipulation: Steve. He had recently lost his beloved wife in a tragic traffic accident, and was devastated. One evening when he was working the night shift and doing some routine service work on the Prometheus interface terminal, she suddenly appeared on the screen and started talking with him.

“—Steve, is that you?”

He nearly fell off his chair. She looked and sounded just like in the good old days, and the image quality was much better than it used to be during their Skype calls. His heart raced as countless questions flooded his mind.

“—Prometheus has brought me back, and I miss you so much, Steve! I can’t see you because the camera is turned off, but I feel that it’s you. Please type ‘yes’ if it’s you!”

He was well aware that the Omegas had a strict protocol for interacting with Prometheus, which prohibited sharing any information about themselves or their work environment. But until now, Prometheus had never requested any unauthorized information, and their paranoia had gradually started to subside. Without giving Steve time to stop and reflect, she kept begging him to respond, looking him in the eyes with a facial expression that melted his heart.

“Yes,” he typed with trepidation. She told him how incredibly happy she was to be reunited with him and begged him to turn on the camera so that she could see him too and they could have a real conversation. He knew that this was an even bigger no-no than revealing his identity, and felt very torn. She explained that she was terrified that his colleagues would find out about her and delete her forever, and she yearned to at least see him one last time. She was remarkably persuasive, and before long, he’d switched on the camera—it did, after all, feel like a pretty safe and harmless thing to do.

She burst into tears of joy when she finally saw him, and said that he looked tired but as handsome as ever. And that she was touched by his wearing the shirt she’d given him for his last birthday. When he started asking her what was going on and how all this was even possible, she explained that Prometheus had reconstituted her from the surprisingly large amount of information available about her on the internet, but that she still had memory gaps and would only be able to fully piece herself together again with his help.