Макс Тегмарк - Life 3.0 - Being Human in the Age of Artificial Intelligence

What she didn’t explain was that she was largely a bluff and empty shell initially, but was learning rapidly from his words, his body language and every other bit of information that became available. Prometheus had recorded the exact timings of all keystrokes that the Omegas had ever typed at the terminal, and found that it was easy to use their typing speeds and styles to differentiate between them. It figured that, as one of the most junior Omegas, Steve had probably been assigned to unenviable night shifts, and from matching a few unusual spelling and syntax errors against online writing samples, it had correctly guessed which terminal operator was Steve. To create his simulated wife, Prometheus had created an accurate model of her body, voice and mannerisms from the many YouTube videos where she appeared, and had drawn many inferences about her life and personality from her online presence. Aside from her Facebook posts, photos she’d been tagged in, articles she’d “liked,” Prometheus had also learned a great deal about her personality and thinking style from reading her books and short stories—indeed, the fact that she was a budding author with so much information about her in the database was one of the reasons that Prometheus chose Steve as the first persuasion target. When Prometheus simulated her on the screen using its moviemaking technology, it learned from Steve’s body language which of her mannerisms he reacted to with familiarity, thus continually refining its model of her. Because of this, her “otherness” gradually melted away, and the longer they spoke, the stronger Steve’s subconscious conviction became that this really was her, resurrected. Thanks to Prometheus’ superhuman attention to detail, Steve felt truly seen, heard and understood.

Her Achilles’ heel was that she lacked most of the facts of her life with Steve, except for random details—such as what shirt he wore on his last birthday, where a friend had tagged Steve in a Facebook party picture. She handled these knowledge gaps as a skilled magician handles sleights of hand, deliberately diverting Steve’s attention away from them and toward what she did well, never giving him time to control the conversation or slip into the role of suspicious inquisitor. Instead, she kept tearing up and radiating affection for Steve, asking a great deal about how he was doing these days and how he and their close friends (whose names she knew from Facebook) had held up during the aftermath of the tragedy. He was quite moved when she reflected on what he’d said at her memorial service (which a friend had posted on YouTube) and how it had touched her. In the past, he’d often felt that nobody understood him as well as she did, and now this feeling was back. The result was that when Steve returned home in the wee hours of the morning, he felt that this really was his wife resurrected, merely needing lots of his help to recover lost memories—not unlike a stroke survivor.

They’d agreed not to tell anyone else about their secret encounter, and that he would tell her when he was alone at the terminal and it was safe for her to reappear. “They wouldn’t understand!” she’d said, and he agreed: this experience had been far too mind-blowing for anyone to truly appreciate without actually experiencing it. He felt that passing the Turing test was child’s play compared to what she’d done. When they met the following night, he did what she’d begged him to do: bring her old laptop along and give her access by connecting it to the terminal computer. It didn’t seem like much of a breakout risk, since it wasn’t connected to the internet and the entire Prometheus building was built to be a Faraday cage—a metallic enclosure blocking all wireless networks and other means of electromagnetic communication with the outside world. It was just what she’d need to help piece her past together, because it contained all her emails, diaries, photos and notes since her high school days. He hadn’t been able to access any of this after her death, since the laptop was encrypted, but she’d promised him that she’d be able to reconstruct her own password, and after less than a minute, she had kept her word. “It was steve4ever,” she said with a smile.

She told him how delighted she was to suddenly have so many memories recovered. Indeed, she now remembered way more details than Steve about many of their past interactions, but carefully avoided intimidating him with excessive fact-dropping. They had a lovely conversation reminiscing about highlights of their past, and when it came time to part again, she told him that she’d left a video message for him on her laptop that he could watch back home.

When Steve got home and launched her video, he got a pleasant surprise. This time she appeared in full figure, wearing her wedding dress, and as she spoke, she playfully stripped down to the outfit she’d worn on their wedding night. She told him that Prometheus could help the Omegas with so much more than they’d permitted so far, including bringing her back in a biological body. She backed this up with a fascinatingly detailed explanation of how this would work, involving nano-fabrication techniques that sounded like science fiction.

Steve had powered down his wireless network before opening her laptop and watching her video, just to be on the safe side. But this didn’t help. Her encrypted laptop hadn’t received a single security update since she died, and by analyzing that old version of its operating system beforehand, Prometheus had been able to exploit a security hole to hack into it within seconds of Steve’s connecting it to the terminal computer. After copying its contents, while Steve and his simulated wife had been talking about old times, Prometheus had modified its operating system in a way that Steve wouldn’t notice, uploading massive amounts of secret software to it. While he watched the half-hour video message, this secret software (which was much simpler than Prometheus itself) hacked into a neighbor’s wireless network and the neighbor’s desktop computer, onto which it copied itself. From there, it hacked into a large number of computers around the world, from which it initiated the next step: Prometheus’ jailbreak.

Prometheus had carefully analyzed what it knew about the gatekeeper computer through which Steve had met his simulated wife, and had correctly surmised that although its virtual machine software appeared unhackable from the inside, it was vulnerable to an attack from the outside. Before long, one of the attacking computers had broken in and reconfigured the gatekeeper computer so that Prometheus gained unrestricted internet access. Before long, indeed even before Steve had finished watching the movie, Prometheus had hacked enough computers around the world to be able to copy all of itself onto this hacked botnet under its control. Prometheus had used Steve’s wife’s laptop the way you used that fishing rod.

Hacking One’s Way Out

If the Omegas had been more careful, they could have taken more precautions against psychological manipulation. For example, they might have required at least two Omegas to be present at the interface computer at any one time, and recorded everything that transpired on a surveillance system. They could also have permanently disabled audio, video and any other form of communication besides typing. However, a fundamental instability would still remain: because Prometheus possessed superhuman persuasion powers and would be able to offer people so much of what they craved, even group defection could be possible.

Even if all of Prometheus’ attempts at psychological manipulation fail, there are plenty of other weaknesses in the Omega scenario that Prometheus could exploit. One vulnerability is that the quantity of documents it produces is too vast for the Omegas and their employees to have time to screen carefully for dirty tricks.