Maureen McHugh - After the Apocalypse

“Then we’ll install from backup,” Damien said. “We’ll do it at 1:00 a.m. We’ll send out a system maintenance bulletin.” Hospitals don’t shut down, but they do a lot less at 1:00 a.m.

Sydney said, “But if we install from backup, we’ll be killing it.”

Damien leaned back in his chair. “Ah, the old transporter question.” They were sitting in his cube. His desk was a mess—stacked with papers, binders, a couple of manuals, and the remains of a dinner of Chinese food. “In Star Trek, if I beam you down to the planet, does that mean I have actually killed you and sent an exact replica in your place?”

“Yes,” Sydney said. She was wearing her If You’re Really a Goth, Where Were You When We Sacked Rome? T-shirt because Damien had laughed his ass off when she first wore it. Damien was wearing cut-off sweatpants and yellow flip-flops because even though the office was technically business casual, no one cared what you wore at 1:00 a.m. “Look, what if we shut it down, back it up, and it never comes back to consciousness?” She was thinking about the book The Moon Is a Harsh Mistress where the AI, Mike, is damaged during the war and after that never speaks again to Manny, the main character. Manny calls the secret phone number he has for Mike, but when he does, there is only silence. She’d read it when she was thirteen, and it had haunted her. She told herself that eventually Mike “woke up” again and called Manny.

“Why do you think it’s conscious?” Damien said.

“Why do I think you’re conscious?” Sydney said.

“You think I’m conscious because I’m like you, and you’re conscious,” Damien said. “DMS isn’t like us.”

“But if it’s aware, then it has consciousness,” Sydney said.

“Is a shark conscious?” Damien said.

“Yeah,” Sydney said.

“How about a cricket? How about a jellyfish? A sponge?”

“If we don’t know if DMS is conscious or not, then we pretty much have to assume it is,” Sydney said. “And if we back it up, we might kill it.”

Damien shook his head. “How can we kill it?”

Sydney said, “Because we will stop it and reinstall it.”

“So you think that the interruption of consciousness might be enough to kill it? You think it has a soul? Its consciousness is in the code. Its code and body are unchanged. If someone has a heart attack and you shock them back, they come back as themselves. Your body is you. DMS’s software and hardware is DMS.” Damien was very pleased with himself.

Sydney was pretty sure it wasn’t so simple. It wasn’t until the next day that she thought of a cogent argument, which was that organic systems are a lot less fragile than computer systems. Organic systems decay gracefully. Computer systems break easily. DMS was much more fragile than an animal. But that night she couldn’t think of anything.

The problem with poking the system to see if it was aware was to figure out what it could sense. DMS didn’t see or hear, didn’t eat or breathe. Its “senses” were all involved in interpreting data. So the “poke” needed to be something that it would recognize, that it would sense. And the poke needed to be something that it would sense as meaningful. The idea that Damien came up with was to feed it information in a way that it could recognize was a pattern but that wasn’t a pattern it expected.

DMS had several systems which regulated input and scanned for patterns. Epidemiological information was generated from ER, patient intake, and pharmaceutical information. Maintenance issues were anticipated from electrical usage. They picked the maintenance system, since DMS had been screwing with the electrical system, and input a thousand-character string of ones and zeroes. It was, Damien said, boring but clearly a pattern.

Sydney wasn’t sure it was the right kind of pattern. “Basically,” she said, “It’s like I flipped a coin and it came up heads a thousand times.”

“Yeah,” Damien said.

“If I did that, I might assume there was something wrong with the coin. But I wouldn’t assume aliens were trying to communicate with me through my coin toss.”

“DMS doesn’t have to recognize that we’re trying to communicate with it,” Damien explained. “It just has to notice that the information is not junk.”

DMS kicked the entry into the garbage column on its maintenance report.

They had written a program to do the entry, so they ran the program a thousand times.

If DMS noticed, it didn’t think anything of it. One thousand times it kicked the entry into the junk portion of the report.

“I don’t think it knows what we’re doing,” Sydney said. “You know, analyzing reports may be unconscious.”

“I don’t think consciousness is an issue here,” Damien said. “Remember the shark.”

“Okay. Maybe it’s involuntary. The shark has control over what fish it goes after, but it doesn’t have control over its kidney function. It doesn’t choose anything about kidney function. Maybe maintenance is involuntary.”

Damien looked at her. She thought he was going to say something dismissive, but after a moment he said, “Well, then, what parts of it would be voluntary?”

Sydney shrugged. “I don’t know. Epidemiologist, maybe. But we can’t screw too much with that.”

Screwing with maintenance was bad enough. But data from LEGBA went directly to the CDC and National Institute of Health through a weird subroutine called DAMBALLAH which did complicated pattern recognition and statistical stuff. Sydney worried about a couple of things. One was causing a system crash that meant someone ended up dying. The other was getting them in trouble with the CDC or the government. Of the two, she would have to choose getting in trouble, except she could imagine bad data to the CDC might mean someone ended up dying anyway. In her mind it unfolded: bad information seems to indicate a critical alert, Marburg virus reports in New York City seem to show that someone got off a plane and infected people with a hemorrhagic fever. The false epidemic pulls resources from a real outbreak of Legionnaires’ disease, and people who would have lived now die because she and Damien were poking DMS.

She thought Damien would say to poke DAMBALLAH. Damien seemed a lot less concerned about getting in trouble than she did. She had a theory that the fear of getting in trouble was what made her not as good a programmer and that, in fact, it was all linked to testosterone, and that was why there were more guy programmers than women. It was a very hazy theory, and she didn’t like it, but she had pretty much convinced herself it was true, although she couldn’t bear to think of sharing it with anybody, because it was a lot better to think that there were social reasons why girls didn’t usually become code monkeys than to think there were biological reasons. But right now she was pretty sure that she would say stop and Damien would say go.

He surprised her. “Not DAMBALLAH. You think that DMS might be fucking with the outputs on DAMBALLAH?”

She shrugged. “I don’t know.”

“Maybe tomorrow we can try to check that.”

Tomorrow was tough, because when Sydney got home she was too keyed up to sleep, and she was up until almost four reading a book called Dead Until Dark . The book had been recommended to her by Addy, her college roommate from junior and senior years. It was the first in a series about a paranormal detective and had been just about the most perfect thing to read after coming home from a failed attempt to prove that a computer system was aware.

She was still worried about DAMBALLAH and whether DMS was doing weird things with the epidemiological reports. DAMBALLAH was a complicated system. It made decisions about reporting data. She couldn’t easily check its decisions—that was the point. Every two weeks they got a report from the NIH and the CDC about epidemiological trends, and if there was something new that the CDC was looking for, say an outbreak of shigella in preschools in the South, there was an elaborate way they entered additional parameters to DAMBALLAH’s tracking system. The CDC and the NIH also sent them error reports and WRs. WRs were to correct when DMS was reporting something that wasn’t important or was overreporting. The result was the DMS “learned” epidemiology.