Bernard Beckett - Genesis

Art held Adam’s stare, and a slow smile spread across his face.

“The difficulty this argument brings,” Art told Adam, “is that from where you stand, this is just how it must appear to you. I am not arguing with your definition, only your contention that I too cannot think by these standards.

“It is natural for you to feel the way you do. You have seen many machines. You have seen them built, and you know they are nothing but moving parts and circuitry. You know they do not think. Automatically opening doors do not think. An oven does not think. A gun has no mind of its own. And so you conclude no machine thinks.

“For you, thought seems to require some extra special substance. But try to see this from my point of view. I see many creatures with brains. A worm perhaps, a fruit fly, a bumblebee. Do they think, or are they just machines?

“I can speak to you in seven languages. I can reason with you in all of them. I can build a version of myself from scratch. I can write poetry, I can beat you at chess. So, who is more of a thinking thing, me or the bumblebee? I am just a machine, while the bumblebee has a brain. So surely by your reasoning, the bumblebee is more of a thinker.”

“My brain is far bigger than that of a bumblebee.”

“My circuitry is far more sophisticated than that of an automatic door.”

They faced one another now, the sort of standoff found in a pre-classical movie, but it teetered on the edge of comedy thanks to the great difference in their heights.

“When I was young, before they moved me on to soldiering, our instructors taught us of a puzzle they called the Chinese Room.”

“I know it well.”

“Am I going to be allowed to tell my story?”

“You know I’ll have an answer for it.”

“When they get around to making more robots,” Adam said, “they’re not going to like you either.” He moved back to his seat.

Art stood before him, waiting for the story to continue. Some of the anger had gone out of Adam now. He spoke slowly as if measuring his words, as if they surprised him, the order in which they tumbled from his mouth.

“In the Chinese Room puzzle,” Adam said, “we are asked to consider a room with a remarkably complex set of levers and pulleys. The most elaborate set you could imagine. Next, we have to suppose that I am seated in the middle of the room, and a message is passed through a slot in the wall, written entirely in a language I do not understand. Chinese, say. Now, the puzzle assumes that I have a book with a long set of instructions, telling me which lever I should push for each character I find written on the note. The pulleys all move, and by observing the movements and following my instruction book, I pull more pulleys, and move more levers, and eventually the levers stop and the machine’s pointing arm indicates a chart on the wall, ticking off the characters I should copy in my reply.

“I do this as the machine instructs and pass my message out the slot. I did not understand the note coming in, and I do not understand the note going out. But, thanks to the intervention of the intricate design of pulleys and levers, the note makes perfect sense to the speaker of Chinese, on the other side of the wall.

“He writes another note, I follow my instructions again and so it goes on. In this way, a conversation takes place between the Chinese speaker and myself. Only, I am not conscious of the content of the messages being passed through the slots. I am involved in a thoughtless conversation.

“The point, as we were taught it, is that there is more to consciousness than mere mechanics. There is a difference between the appearance of thought, and thought itself. The Chinese speaker assumes there is a thinking entity on the other side of the wall with whom they’re conversing, but this assumption is quite wrong. There is only a collection of pulleys and levers, and me, at the heart of it, following instructions, understanding nothing. And that’s what I think you are. I think you are the Chinese Room.”

“I think I am the Chinese Room too/’ Art replied. “And that is what is wrong with your example.”

Adam looked to Art, waiting for his explanation. “I don’t understand.” They were quieter now, more respectful. As if they knew they were approaching this place together, and that once there, there would be no turning back.

“I could explain it to you,” Art told him, speaking gently now, staring deep into Adam’s eyes, “but I do not think you will want to hear it. You are too clever to ignore a good explanation, and then you won’t be able to treat me as a machine any more. That will be very hard for you. So perhaps I should wait until you are ready to hear it. Perhaps, if I wait long enough, you will work it out for yourself.”

“It’s your decision,” Adam told him.

“No,” Art insisted. “I want you to decide.”

“Give me your explanation.”

“Are you sure?”

Adam hesitated. “I am sure.”

“All right,” Art nodded. “The first message the Chinese speaker writes is, ‘I’m going to burn your building down.’ Now, tell me, what does the machine reply?”

“It’s not important,” Adam said. “Just so long as it makes sense. That’s all the problem requires.”

“No,” Art corrected. “It requires something more. There is an endless choice of sensible responses. It could call the bluff with, ‘Please do, I’m sick of being trapped in here.’ It could try aggression: ‘Don’t make me come out there and whip your Chinese- speaking ass.’ It could try to distract: ‘Why do you want to set light to me?’ Or how about pleading? ‘Please no, I’ll do anything. Name your price.’ A thousand things to say, and for each a million ways of expressing them. Your example works only if we can imagine how the machine chooses its response.”

“I don’t think it matters how it does it. Just say it chooses one at random. The first one that comes to mind.”

“But it doesn’t have a mind.”

“It isn’t meant to be real.” Adam was growing frustrated. “That’s not the point. It’s demonstrating a principle.”

“Yes,” Art allowed, “but think about the principle a little more deeply. You told me before that you are different from me because you make meaning of things. But look at what your room must be able to do. It must be able to interpret the intentions of the Chinese speaker, and it must be able to pursue its own objectives in framing its responses. If it has no intentions, it can make no conversation.”

“Not true,” Adam interrupted. “It might simply be a system programmed to interpret patterns. When this symbol shows, print that symbol. If the program is complex enough that might fool the speaker.”

“That rather depends upon the intelligence of the speaker, but we are missing the point. For a simple conversation, of course the room does not have to be conscious, any more than you have to engage your consciousness to grunt your greetings to the guards who clean out your cell. But at some point, when the room is called upon to access its own memories, respond to changing circumstances, modify its own objectives, all the things you do when you engage in a meaningful conversation, all that changes. You think the thing you call consciousness is some mysterious gift from the heavens, but in the end consciousness is nothing but the context in which your thinking occurs. Consciousness is the feel of accessing memory. Why else do you not have memories from your earliest years? It is because your consciousness has not fully developed.”

“You’re avoiding the question,” Adam insisted, but there was doubt in his eyes. “I’m in the room and I do not understand the conversation at all. The conversation takes place, even though I am not conscious of it. Explain that, if you can.”