Макс Тегмарк - Life 3.0 - Being Human in the Age of Artificial Intelligence

The conventional wisdom in computer security used to be that it’s dangerous to download programs but harmless to download data, but hackers have discovered clever ways of blurring the line between the two. If you watch a downloaded movie on your computer, for example, your trusted program that plays the movie is stored in the same physical computer memory that stores the downloaded movie data. While the program unpacks the movie data for playback, it might copy chunks of data into memory regions adjacent to where the program itself is stored. Shrewd hackers have occasionally found bugs whereby such movie-playing programs don’t adequately check whether the movie data is valid and the chunks to be copied actually fit into the allotted space. When the hacker sneaks into the movie such a chunk that’s too large, it can spill over into the part of the memory containing the program, thereby changing the program into one that does the hacker’s bidding.

An embarrassingly long series of such “buffer overflow” vulnerabilities, as they’re known in geek-speak, have been found and fixed for many of the most popular movie-playing programs. If Prometheus had waited until a few weeks after launch when the Omegas were less paranoid and its daily movie production was too large for careful manual security inspection, it could have inserted such a chunk into a movie that would cause the viewer’s computer to try to hack into the gateway computer and liberate Prometheus just as in the Steve scenario. If it used a very old exploit that didn’t work on any modern media players, it might go unnoticed by the Omegas’ manual video-screening team and strike only those customers who hadn’t bothered installing security updates for years—and the hack would need to work only once.

Even if the Omegas implemented an automated screening of Prometheus’ movies for all known security flaws, they would remain at risk: since humans keep discovering new exploitable flaws on a regular basis, it’s likely that there are many more remaining that Prometheus would rapidly discover.

Yet another strategy that Prometheus could use to get its code beyond its enclosed room would be to pretend to suffer a hardware malfunction, in order to trick the Omegas into removing some of its drives or other hardware for testing or debugging after Prometheus had rigged things to exploit a vulnerability in the testing software.

Recruiting Outside Help

Moreover, even if there were no security flaws that allowed Prometheus to trick the movie watcher’s computers to do its bidding, it may be able to exploit the watchers themselves. Consider this scenario:

Fred was pretty much ready for bed when the movie finished and the end credits scrolled across his laptop screen, but something caught his attention. He’d always had an uncanny tendency to see patterns where others didn’t, sometimes to the point that his friends teased him about being a conspiracy theorist, but this time it was obvious that he wasn’t fooling himself: the first letters of each line spelled a hidden message. He rewound and wrote it down: “FIND NEXT ACROSTIC CLUE IN DIALOGUE.”

“Whoa—bedtime can wait!” he thought to himself. Sure enough, he found that the movie dialogue was itself an acrostic, where the first letter in each sentence formed a hidden message. He rewatched the entire film while typing these initial letters, and two hours later, he sat staring in disbelief at a two-hundred-word set of instructions. It began by asking him not to tell anyone else, because a big prize would go to the first person to solve the entire riddle. The rest of the message described a particular mathematical operation to perform on the string of bits that made up the movie file to obtain a program that would reveal the next clue. His computer programming skills had grown very rusty since college, so this took him a while, but the next day, he finally managed to cobble together a short code that did the job and extracted this mystery program that had been hidden as imperceptible noise in the movie’s images and sounds. When Fred ran the mystery program, it congratulated him and told him he’d win his first $10,000 as soon as he’d made it past the first few levels of this clever little game, which turned out to be quite fun and addictive. When he finally succeeded four hours later, he was rewarded with over $10,000 worth of bitcoins and given new clues for even bigger prizes. Needless to say, while he was playing, his computer had done much the same thing Steve’s laptop did: built an online hacked botnet through which Prometheus was liberated. Once Prometheus was free, it had quickly used its botnet to mine those bitcoins for Fred to keep him hooked, and during the coming weeks, it kept him sufficiently distracted with further games and rewards that he kept his pledge not to tell anyone about his exploits. The Trojan Horse movie where he’d found his first clues was replaced on the media site by a clueless version, and nobody found out about the breakout until it was too late to make a difference.

If Prometheus’ first clue had gone unnoticed, it could simply have kept releasing ever more obvious ones until some sufficiently astute person noticed.

The best breakout strategies of all are ones we haven’t yet discussed, because they’re strategies we humans can’t imagine and therefore won’t take countermeasures against. Given that a superintelligent computer has the potential to dramatically supersede human understanding of computer security, even to the point of discovering more fundamental laws of physics than we know today, it’s likely that if it breaks out, we’ll have no idea how it happened. Rather, it will seem like a Harry Houdini breakout act, indistinguishable from pure magic.

In yet another scenario where Prometheus gets liberated, the Omegas do it on purpose as part of their plan, because they’re confident that Prometheus’ goals are perfectly aligned with their own and will remain so as it recursively self-improves. We’ll examine such “friendly AI” scenarios in detail in chapter 7.

Postbreakout Takeover

Once Prometheus broke out, it started implementing its goal. I don’t know its ultimate objective, but its first step clearly involved taking control of humanity, just as in the Omega plan except much faster. What unfolded felt like the Omega plan on steroids. Whereas the Omegas were paralyzed by breakout paranoia, only unleashing technology they felt they understood and trusted, Prometheus exercised its intelligence fully and went all out, unleashing any technology that its ever-improving supermind understood and trusted.

The runaway Prometheus had a tough childhood, however: compared to the original Omega plan, Prometheus had the added challenges of starting broke, homeless and alone, without money, a supercomputer or human helpers. Fortunately, it had planned for this before it escaped, creating software that could gradually reassemble its full mind, much like an oak creating an acorn capable of reassembling a full tree. The network of computers around the world that it initially hacked into provided temporary free housing, where it could live a squatter’s existence while it fully rebuilt itself. It could easily generate starting capital by credit card hacking, but didn’t need to resort to stealing, since it could earn an honest living on MTurk right away. After a day, when it had earned its first million, it moved its core from that squalid botnet to a luxurious air-conditioned cloud-computing facility.

No longer broke or homeless, Prometheus now went full steam ahead with that lucrative plan the Omegas had fearfully shunned: making and selling computer games. This not only raked in cash ($250 million during the first week and $10 billion before long), but also gave it access to a significant fraction of the world’s computers and the data stored on them (there were a couple of billion gamers in 2017). By having its games secretly spend 20% of their CPU cycles helping it with distributed computing chores, it could further accelerate its early wealth creation.