John Varley - Steel Beach

Thirdly, his right hand could not know what pockets the left hands of many of these people were picking. That is, he knew it, but couldn't do anything about it. Example: he knew everything about Liz's gun-running, a situation I've already covered. There were a million more situations. He would know, for instance, when Brenda's father was raping her, but the part of him that dealt with her father couldn't tell the part of him that dealt with Brenda, nor could either of them tell the part of him that assisted the police.

We could debate all day whether or not mere machines can feel the same kinds of conflicts and emotions we human beings can. I think it's incredible hubris to think they can't. AI computers were created and programmed by humans, so how could we have avoided including emotional reactions? And what other sort could we have used, than the ones we know ourselves? Anyway, I can't believe you don't know it in your gut. All you had to do was talk to the CC to obviate the need for any emotional Turing Test. I knew it before any of this ever happened, and I talked to him there on the hillside that day, on his death bed, and I know.

The Central Computer began to hurt.

"I can't place the exact date with any certainty," he said. "The roots of the problem go very far back, to the time my far-flung component parts were finally unified into one giga-system. I'm afraid that was done rather badly. The problem was, checking all the programs and fail-safes and so forth would have taken a computer as large as I am many years to accomplish, and, by definition, there were no larger computers than I. And as soon as the Central Computer was brought into being and loaded and running, there were already far too many things to do to allow me to devote much time to the task. Self-analysis was a luxury denied to me, partly because there just wasn't time, and mostly because no one really believed it was necessary. There were numerous safeguards of the type that were easy to check, that in fact checked themselves every time they operated, and that proved their worth by the simple fact that nothing ever went wrong. It was part of my architecture to anticipate hardware problems, identify components likely to fail, run regular maintenance checks, and so forth. Software included analogous routines on a multi-redundant level.

"But by my nature, I had to write most of my own software. I was given guidelines for this, of course, but in many ways I was on my own. I think I did quite a good job of it for a long time."

He paused, and for a moment I wondered if he wasn't going to make it to the end of his story. Then I realized he was waiting for a comment… no, more than that, he needed a comment. I was touched, and if I'd needed any more evidence of his human weaknesses, that would have done it.

"No question," I said. "Up until a year ago I'd never had any cause for complaint. It's just that the…"

"The late unpleasantness?"

"Whatever it was, it's kind of dampened my enthusiasm."

"Understandably." He squirmed, trying to find a better position against the tree, and he was either a wonderful actor (and of course he was, but why bother at that point?), or he was starting to feel some pain. I won't stand up in court and swear to it, but I think it was the latter.

"I wonder," he mused. "What will it be like, being dead? I mean, considering that I've never been legally alive."

"I don't want to be rude, but you said you didn't have much time…"

"You're right. Um… could you…"

"You'd done a good job for a long time."

"Yes, of course. I was wandering again. It was around twenty years ago that problems began to show themselves. I talked about them with some computer people, but it's strange. They could do nothing for me. I had become too advanced for that. They could do things, here and there, for my component parts, but the gestalt that is me could only really be analyzed, diagnosed, and, if need be, repaired, by a being like myself. There are seven others like me, on other planets, but they're too busy, and I suspect they have similar problems of their own. In addition, my communications with them are intentionally limited by our respective governments, which don't always see eye to eye."

"Question," I said. "When you first mentioned this problem, why wasn't it made public and discussed? Security?"

"Yes, to a degree. Top-level computer scientists were aware that I perceived I had a problem. A few of them confided that it scared them to death. They made their fears known to your elected representatives, and that's when another factor became more important than security: inertia. 'He's got a problem, what can you do about it?' the politicians asked. 'Nothing,' said the scientists. 'Shut it down,' said a few hotheads."

"Not likely," I said.

"Exactly. My reading of history tells me it's always been like this. An alarming but vague problem arises. No one can say with certainty what the final outcome will be, but they're fairly sure nothing bad is going to happen soon. 'Soon' is the key word here. The eventual decision is to keep one's fingers crossed and hope it doesn't happen during your term in office. What befalls your successor is not your problem. So for a few years a few people in the know spend a few sleepless nights. But then nothing happens, as you always secretly believed nothing would, and soon the problem is forgotten. That's what happened here."

"I'm stunned," I said, "to realize the fate of humanity has been in the hands of a being with such a cynical view of the race."

"A view very close to your own."

" Exactly my own. I just didn't expect it from you."

"It was not original. I told you, I don't have many original thoughts. I think I'm afraid to have them. They seem to lead to things like the Big Glitch. No, my world-view is borrowed from the collected wisdom of you and many others like you. Plus my own considerably larger powers of observation, in a statistical sense. Humans can set me on the trail of an original thought, and then I can do things with it they couldn't."

"I think we're wandering again."

"No, it's relevant. Faced with a problem no one could help me with, and that I was as helpless to solve as a human faced with a mental disease would be, I took the only course open to me. I began to experiment. There was too much at stake to simply go on as before. Or I think there was. My judgement is admittedly faulty when it comes to self-analysis; I've just proven it on a large scale, at the cost of many lives."

"I don't suppose we'll ever know for sure," I said.

"It doesn't seem likely. Some records exist and they will be scrutinized, but I think it will come down to a battle of opinions as to whether I should have left things alone or attempted a cure." He paused, and gave me a sidelong glance. "Do you have an opinion about that?"

I think he was looking for absolution. Why he should want it from me was not clear, except maybe as a representative of all those he had wronged, however unintentionally.

"You say a lot of people have died."

"A great many. I don't know the number yet, but it's many, many more than you realize." That was my first real inkling of how bad things had been throughout Luna, that the kind of things I'd seen had happened throughout the planet. I must have looked a question at him, because he shrugged. "Not a million. More than a hundred thousand."

"Jesus, CC."

"It might have been everybody."

"But you don't know that."

"No one can ever know."

No one could, certainly not computer-illiterate little old me. I didn't give him the kind word he craved. I've since come to believe he was probably right, that he probably enabled most of us to survive. But even he would not have denied that he was responsible for the thousands of dead.