Peter Watts - Starfish

"You're kidding. You're fucking kidding ."

"— so they gave the keys to a smart gel. Even that was problematic, actually. They had to pull one out of the net at random so no one could claim it'd been preconditioned, and every member of the consortium had to have a hand in team-training it. Then there was the question of authorizing it to take— necessary steps, autonomously…"

"You gave control to a smart gel? A head cheese ?"

"It was the only way."

"Rowan, those things are alien !"

She grunted. "Not as alien as you might think. The first thing this one did was get another gel installed down on the rift, running simulations. We figured under the circumstances, nepotism was a good sign."

"They're black boxes, Rowan. They wire up their own connections, we don't know what kind of logic they use."

"You can talk to them. If you want to know that sort of thing, you just ask."

" Jesus Christ! " Scanlon put his face in his hands, took a deep breath. "Look. For all we know these gels don't understand the first thing about language."

"You can talk to them." Rowan was frowning. "They talk back."

"That doesn't mean anything. Maybe they've learned that when someone makes certain sounds in a certain order, they're supposed to make certain other sounds in response. They might not have any concept at all of what those sounds actually mean . They learn to talk through sheer trial and error."

"That's how we learn too," Rowan pointed out.

" Don't lecture me in my own field! We've got language and speech centers hardwired into our brains. That gives us a common starting point. Gels don't have anything like that. Speech might just be one giant conditioned reflex to them."

"Well," Rowan said. "So far it's done its job. We have no complaints."

"I want to talk to it," Scanlon said.

"The gel?"

"Yes."

"What for?" She seemed suddenly suspicious.

"You know me. I specialize in aliens."

Rowan said nothing.

"You owe me this, Rowan. You fucking owe me. I've been a faithful dog to the GA for ten years now. I went down to the rift because you sent me, that's why I'm a prisoner now, that's why— this is the least you can do."

Rowan stared at the floor. "I'm sorry," she muttered. "I'm so sorry."

And then, looking up: "Okay."

It only took a few minutes to establish the link.

Patricia Rowan paced on her side of the barrier, muttering softly into a personal mike. Yves Scanlon sat slumped in a chair, watching her. When her face fell into shadow he could see her contacts, glittering with information.

"We're ready," she said at last. "You won't be able to program it, of course."

"Of course."

"And it won't tell you anything classified."

"I won't ask it to."

"What are you going to ask it?" Rowan wondered aloud.

"I'm going to ask it how it feels," Scanlon said. "What do you call it?"

"Call it?"

"Yes. What's its name?"

"It doesn't have a name. Just call it gel ." Rowan hesitated a moment, then added, "We didn't want to humanize it."

"Good idea. Hang on to that common ground." Scanlon shook his head. "How do I open the link?"

Rowan pointed at one of the touch screens embedded in the conference table. "Just activate any of the panels."

He reached out and touched the screen in front of his chair. "Hello."

"Hello," the table replied. It had a strange voice. Almost androgynous.

"I'm Dr. Scanlon. I'd like to ask you some questions, if that's okay."

"That's okay," the gel said after a brief hesitation.

"I'd like to know how you feel about certain aspects of your, well, your job."

"I don't feel," said the gel.

"Of course not. But something motivates you, in the same way that feelings motivate us. What do you suppose that is?"

"Who do you mean by us?"

"Humans."

"I'm especially likely to repeat behaviors which are reinforced," the gel said after a moment.

"But what motivates— no, ignore that. What is most important to you?"

"Reinforcement is important, most."

"Okay," Scanlon said. "Does it feel better to perform reinforced behaviors, or unreinforced behaviors?"

The gel was silent for a moment or two. "Don't get the question."

"Which would you rather do?"

"Neither. No preference. Said that already."

Scanlon frowned. Why the sudden shift in idiom?

"And yet you're more likely to perform behaviors that have been reinforced in the past," he pressed.

No response from the gel. On the other side of the barrier Rowan sat down, her expression unreadable.

"Do you agree with my previous statement?" Scanlon asked.

"Yeah," drawled the gel, it's voice edging into the masculine.

"So you preferentially adopt certain behaviors, yet you have no preferences."

"Uh huh."

Not bad. It's figured out when I want confirmation of a declarative statement. "Seems like a bit of a paradox," Scanlon suggested.

"I think that reflects an inadequacy in the language as spoken." That time, the gel almost sounded like Rowan.

"Really."

"Hey," said the gel. "I could explain it to you if you wanted. Could piss you off though."

Scanlon looked at Rowan. Rowan shrugged. "It does that. Picks up bits and pieces of other people's speech patterns, mixes them up when it talks. We're not really sure why."

"You never asked?"

"Someone might have," Rowan admitted.

Scanlon turned back to the table. "Gel, I like your suggestion. Please explain to me how you can prefer without preference."

"Easy. Preference describes a tendency to… invoke behaviors which generate an emotional payoff. Since I lack the receptors and chemical precursors essential to emotional experience, I can't prefer . But there are numerous examples… of processes which reinforce behavior, but which… do not involve conscious experience."

"Are you claiming to not be conscious?"

"I'm conscious."

"How do you know?"

"I fit the definition." The gel had adopted a nasal, sing-song tone that Scanlon found vaguely irritating. "Self-awareness results from quantum interference patterns inside neuronal protein microtubules. I have all the parts. I'm conscious."

"So you're not going to resort to the old argument that you know you're conscious because you feel conscious."

"I wouldn't buy it from you."

"Good one. So you don't really like reinforcement?"

"No."

"Then why change your behavior to get more of it?"

"There… is a process of elimination," the gel admitted. "Behaviors which aren't reinforced become extinct. Those which are, are… more likely to occur in the future."

"Why is that?"

"Well, my inquisitive young tadpole, reinforcement lessens the electrical resistance along the relevant pathways. It just takes less of a stimulus to evoke the same behavior in future."

"Okay, then. As a semantic convenience, for the rest of our talk I'd like you to describe reinforced behaviors by saying that they make you feel good , and to describe behaviors which extinguish as making you feel bad . Okay?"

"Okay."

"How do you feel about your present functions?"

"Good."

"How do you feel about your past role in debugging the net?"

"Good."

"How do you feel about following orders?"

"Depends on order. Good if promotes a reinforced behavior. Else bad."

"But if a bad order were to be repeatedly reinforced, you would gradually feel good about it?"

"I would gradually feel good about it," said the gel.

"If you were instructed to play a game of chess, and doing so wouldn't compromise the performance of your other tasks, how would you feel?"