Douglas Hofstadter - The Mind’s I - Fantasies and Reflections on Self and Soul

CHRIS: More exactly put, you can hypothesize about what kinds of internal structures might account for the behaviour observed—but they may or may not in fact exist.

SANDY: Hold on, now! Are you saying that atomic nuclei are merely hypothetical entities? After all, their existence—or should I say “hypothetical existence”?—was proven, or should I say “suggested”?—by the behaviour of particles scattered off atoms.

CHRIS: Physical systems seem to me to be much simpler than the mind, and the certainty of the inferences made is correspondingly greater.

SANDY: The experiments are also correspondingly harder to perform and to interpret. In the Turing test, you could perform many highly delicate experiments in the course of an hour. I maintain that people give other people credit for being conscious simply because of their continual external monitoring of them—which is itself something like a Turing test.

PAT: That may be roughly true, but it involves more than just conversing with people through a teletype. We see that other people have bodies, we watch their faces and expressions—we see they are fellow human beings and so we think they think.

SANDY: To me, that seems like a highly anthropocentric view of what thought is. Does that mean you would sooner say a mannikin in a store thinks than a wonderfully programmed computer, simply because the mannikin looks more human?

PAT: Obviously I would need more than just vague physical resemblances to the human form to be willing to attribute the power of thought to an entity. But that organic quality, the sameness of origin, undeniably leads a degree of credibility that is very important.

SANDY: Here we disagree. I find this simply too chauvinistic. I feel that the key thing is a similarity of internal structure—not bodily, organic, chemical structure, but organizational structure—software. Whether an entity can think seems to me to be a question of whether its organization can be described in a certain way, and I’m perfectly willing to believe that the Turing test detects the presence or absence of that mode of organization. I would say that your depending on my physical body as evidence that I am a thinking being is rather shallow. The way I see it, the Turing test looks far deeper than at mere external form.

PAT: Hey now—you’re not giving me much credit. It’s not just the shape of a body that lends weight to the idea that there’s real thinking going on inside—it’s also, as I said, the idea of common origin. It’s the idea that you and I both sprang from DNA molecules, an idea to which I attribute much depth. Put it this way: The external form of human bodies reveals that they share a deep biological history, and that it’s that depth that lends a lot of credibility to the notion that the owner of such a body can think.

SANDY: But that is all indirect evidence. Surely you want some direct evidence? That is what the Turing test is for. And I think it is the only way to test for “thinkinghood.”

CHRIS: But you could be fooled by the Turing test, just as an interrogator could think a man was a woman.

SANDY: I admit, I could be fooled if I carried out the test in too quick or too shallow a way. But I would go for the deepest things I could think of.

CHRIS: I would want to see if the program could understand jokes. That would be a real test of intelligence.

SANDY: I agree that humour probably is an acid test for a supposedly intelligent program, but equally important to me—perhaps more so—would be to test its emotional responses. So I would ask it about its reactions to certain pieces of music or works of literature—especially my favourite ones.

CHRIS: What if it said, “I don’t know that piece,” or even “I have no interest in music”? What if it avoided all emotional references?

SANDY: That would make me suspicious. Any consistent pattern of avoiding certain issues would raise serious doubts in me as to whether I was dealing with a thinking being.

CHRIS: Why do you say that? Why not say you’re dealing with a thinking but unemotional being?

SANDY: You’ve hit upon a sensitive point. I simply can’t believe that emotions and thought can be divorced. Put another way, I think that emotions are an automatic by-product of the ability to think. They are implied by the very nature of thought.

CHRIS: Well, what if you’re wrong? What if I produced a machine that could think but not emote? Then its intelligence might go unrecognized because it failed to pass your kind of test.

SANDY: I’d like you to point out to me where the boundary line between emotional questions and unemotional ones lies. You might want to ask about the meaning of a great novel. This requires understanding of human emotions? Is that thinking or merely cool calculation? You might want to ask about a subtle choice of words. For that you need an understanding of their connotations. Turing uses examples like this in his article. You might want to ask it for advice about a complex romantic situation. It would need to know a lot about human motivations and their roots. Now if it failed at this kind of task, I would not be much inclined to say that it could think. As far as I am concerned, the ability to think, the ability to feel, and consciousness are just different facets of one phenomenon, and no one of them can be present without the others.

CHRIS: Why couldn’t you build a machine that could feel nothing, but that could think and make complex decisions anyway? I don’t see any contradiction there.

SANDY: Well, I do. I think that when you say you are visualizing a metallic, rectangular machine, probably in an air-conditioned room—a hard, angular, cold object with a million coloured wires inside it, a machine that sits stock still on a tiled floor, humming or buzzing or whatever, and spinning its tapes. Such a machine can play a good game of chess, which, I freely admit, involves a lot of decision making. And yet I would never call such a machine conscious.

CHRIS: How come? To mechanists, isn’t a chess-playing machine rudimentarily conscious?

SANDY: Not to this mechanist. The way I see it, consciousness has got to come from a precise pattern of organization—one that we haven’t yet figured out how to describe in any detailed way. But I believe we will gradually come to understand it. In my view consciousness requires a certain way of mirroring the external universe internally, and the ability to respond to that external reality on the basis of the internally represented model. And then in addition, what’s really crucial for a conscious machine is that it should incorporate a well-developed and flexible self-model. And it’s there that all existent programs, including the best chess-playing ones, fall down.

CHRIS: Don’t chess programs look ahead and say to themselves as they’re figuring out their next move, “If you move here, then I’ll go there, and if you go this way, I could go that way…”? Isn’t that a sort of self model?

SANDY: Not really. Or, if you want, it’s an extremely limited one. It’s an understanding of self only in the narrowest sense. For instance, a chess-playing program has no concept of why it is playing chess, or the fact that it is a program, or is in a computer, or has a human opponent. It has no ideas about what winning and losing are, or —

PAT: How do you know it has no such sense? How can you presume to say what a chess program feels or knows?

SANDY: Oh, come on! We all know that certain things don’t feel anything or even know anything. A thrown stone doesn’t know anything about parabolas, and a whirling fan doesn’t know anything about air. It’s true I can’t prove those statements, but here we are verging on questions of faith.