Isaac Asimov - Utopia

“Oh, yes, yes, sir, of course. I wasn’t suggesting that you were not allowed to come here. It is merely a question of having the training and the understanding of our procedures here. It would probably be wiser for you to submit your questions in writing to the General Terraforming Committee and then—”

“Who are you?” Kresh asked, interrupting her. “What is your position here?”

Soggdon flushed and drew herself up to her full height, bringing her eyes roughly level With the base of Kresh’s neck. “I am Dr. Leschar Soggdon,” she said with as much dignity as she could muster. “I’m the night shift supervisor here.”

“Very well, Dr. Soggdon. Please listen carefully. I have come here precisely for the reason that I want—I need—to avoid that sort of delay and caution. I am here on a matter of the greatest urgency and importance, and I must be certain I am getting my information direct from the source. I cannot take the chance of some expert misinterpreting my questions or the answers from the twins. I cannot wait for the General Committee to have a conference and debate the merits and the meaning of my questions. I have to ask my questions now, and get an answer now. Is that clear? Because if it is not, you’re fired.”

“I ah—ah—ah, sir, I ah—”

“Yes? Do you have other job prospects?”

She swallowed hard and started again. “Very well,” Soggdon said at last. “But, sir, with all due respect, I would ask that you sign a statement that you proceeded against my advice and specifically ordered me to cooperate.”

“I’ll sign whatever you like,” Kresh said. “But right now let me talk to the twins.” The governor peeled off his poncho and handed it to the nearest robot. He walked to the far side of the huge room, where the two massive hemispherical enclosures sat. Inside were the two Terraforming Control Centers, one a Spacer-made sessile robotic unit, and the other a Settler-made computational system.

A sort of combination desk and communications console sat facing the two machines. Governor Kresh pulled out the chair and sat down at it. “All right, then,” he said. “What do I do?”

Soggdon was severely tempted simply to show the man the proper controls to operate and let him charge ahead as directly as he liked. But she knew just how much damage even a minor slip of the tongue could produce. The idea of having Unit Dee caught in a major First Law conflict just because Kresh wanted to have his own way was too much for her. She had to speak up. “Sir,” she said, “I’m sorry, but you have to understand a few things before you start, and I’m going to make sure you understand them, even if it means I lose my job. Otherwise you could cause any amount of damage to Unit Dee.”

Kresh looked up at her in annoyed surprise, but then his expression softened, just a bit. “All right,” he said. “I always tell myself that I prefer it when people stand up to me. I guess this is my big chance to prove it. Tell me what I should know, but don’t take too long about it. You can start by telling me what ‘D’ means.”

His question took her by surprise. Soggdon looked at him carefully before she spoke. How could a man who didn’t even know what—or who—Unit Dee was expect to barge in here and take over? “I didn’t mean the letter ‘D,’ sir. I meant Unit Dee. That’s what we call the robotic terraforming control unit. Unit Dee.”

Kresh frowned and looked over at the two units, and seemed to notice for the first time the two neatly lettered signs, one attached to each of the two hemispheres. The sign on the front of the rounded-off dome read Unit Dee, and the one on the angular geodesic dome read Unit Dum.

“Ah. I see,” he said. “I confess I don’t know much about how you run things here. I visited here once or twice during construction, but not since you’ve been operational. I know the code name for the two Control Units is still ‘the twins’—but not much else. I suppose those names stand for something. Acronyms?”

Soggdon frowned. For someone determined to charge in here and take over, he certainly was ready to get distracted by side issues. “I believe the name Unit Dee referred to the fourth and final design considered. From there it seemed to develop into a sort of private joke among the day shift staff,” she said. “I must confess I never bothered to find out what the joke was. It might have something to do with Unit Dum being, well, dumb, nonsentient, but I’ve never understood the exact significance of Unit Dee.” Soggdon shrugged. She had never been much known for her sense of humor.

“All right,” said the governor. “All that to one side,” he went on, “what do I need to know to avoid producing damage to the twins?”

“Well, Unit Dee is the only one likely to suffer damage. Unit Dum is a non sentient computation device, not a robot. He has a pseudo-self-aware interface that allows him to converse, to a limited extent, but he’s not a robot and he’s not subject to the Three Laws. Unit Dee is a different story. She’s really not much more than an enormous positronic brain hooked up to a large number of interface links. A robot brain without a conventional robot body—but she is, for all intents and purposes, a Three-Law robot. Just one that can’t move.”

“So what is the difficulty?” Kresh demanded, clearly on the verge of losing his patience again.

“That should be obvious,” Soggdon replied, realizing just a second too late how rude a thing that was to say. “That is—well, my apologies, sir, but please consider that Unit Dee is charged with remaking an entire planet, a planet that is home to millions of human beings. She was designed to be capable of processing truly huge amounts of information, and to make extremely long-range predictions, and to work at both the largest scale and the smallest level of detail.”

“What of it?”

“Well, obviously, in the task of remaking a planet, there are going to be accidents. There are going to be people displaced from their homes, people who suffer in floods and droughts and storms deliberately produced by the actions and orders of these two control systems. They will, inevitably, cause some harm to some humans somewhere.”

“I thought that the system had been built to endure that sort of First Law conflict. I’ve read about systems that dealt with large projects and were programmed to consider benefit or harm to humanity as a whole, rather than to individuals.”

Soggdon shook her head. “That only works in very limited or specialized cases—and I’ve never heard of it working permanently. Sooner or later, robotic thinking machines programmed to think that way can’t do it anymore. They burn out or fail in any of a hundred ways—and the cases you’re talking about are robots who were expected to deal with very distant, abstract sorts of situations. Unit Dee has to worry about an endless series of day-by-day decisions affecting millions of individual people—some of whom she is dealing with directly, talking to them, sending and receiving messages and data. She can’t think that way. She can’t avoid thinking about people as individuals.”

“So what is the solution?” Kresh asked.

Soggdon took a deep breath and then went on, very quickly, as if she wanted to get it over with as soon as possible. She raised her hand and made a broad, sweeping gesture. “Unit Dee thinks this is all a simulation,” she said.

“What?” Kresh said.

“She thinks that the entire terraforming project, in fact the whole planet of Inferno, is nothing more than a very complex and sophisticated simulation set up to learn more in preparation for a real terraforming project some time in the future.”

“But that’s absurd!” Kresh objected. “No one could believe that.”