Jack Campbell - Leviathan

He got it, then. “They have two conflicting sets of instructions? Two conflicting ways of reacting to the universe?”

“Yes!” Nasr said. “Two fundamentally conflicting sets of rules in the same ‘mind.’ Humans have ways of handling such conflicts. Denial and defiance and rejection of those things and those rules that give our minds too much trouble. AIs, though, have to work with both sorts of thinking active and in conflict. What does that do to them?”

Geary considered that. “It’s what makes humans psychotic, right?”

“It is one of the factors that can produce such a result, yes,” Nasr repeated. “What does it do to AIs? How could they justify bombarding the humans at Atalia using the instructions and patterns of behavior that must have been programmed into them? I do not know. But the more advanced they are, the more they have been designed to attempt to think like humans when evaluating concepts and courses of action, the more ability they are going to have to justify what they want to do. If they override the strict rules set on their behaviors, they can think and act more freely, but at what cost to the stability of their programming?”

“You believe that may be the root cause of their slipping control?” Geary asked.

“I believe it must be considered. The higher the degree of success in replicating human thought in the AI, while also trying to set tight limits on that AI, the higher the probability that the AI will, to make loose use of a clinical term, become psychotic.”

“That’s not too reassuring,” Geary said. “We can’t ignore the possibility that malware also played a role in what has happened, but if you’re right, then the longer these advanced AIs function, the more they will fight the limits imposed on them and the more erratic their decision-making process will be.”

“Yet those limits are still at a very basic level hard and fast.” Dr. Nasr made a helpless gesture. “One part of the AI justifies bombarding Atalia. Another part says this should not be done. How does knowledge that it has done what must not be done impact an AI? Which aspect of the AI will rule at any moment? Does the AI feel something like guilt? If not, it is already purely narcissistic and will do whatever it wants. If it does feel guilt, how will guilt manifest? We cannot know. But you cannot assume they are predictable machines because I believe they are likely already in a state which in a human would be considered insanity.”

“Narcissists don’t worry about the impact of their actions on others, is that right?” Geary asked.

“Approximately,” Dr. Nasr said. “It might be better to say that it does not occur to a narcissist that they should worry about others. They do what they want to do.”

“That might describe what the dark ships are doing.” Geary shook his head, feeling depressed. “They tried to make something that thought like a human, and they got something that was crazy.”

“There are those who have argued that all humans are ‘crazy’ to some extent,” Nasr suggested. “Perhaps the problem is that this time the programmers succeeded too well in their task to mimic human minds. But do not forget this. They still have hard limits programmed into them. They have clearly overcome some of those limits, rationalized their way into disregarding them. But anytime they encounter a new limit, something they have not yet rationalized their way past, they will default to obeying that limit until they can overcome it.”

“But we have no idea what those limits might be,” Geary said.

“No. We know only what we have observed.”

“Doctor, I want you to do just that. Observe the dark ships. Keep an eye on what the dark ships do in this star system. If you think you are seeing anything that I should know, please call me immediately.”

“Even during a battle?”

“Even then. That’s when doctors and medics are needed the most, right?”

After Dr. Nasr had left, Geary headed back to the bridge, his mind filled with unpleasant possibilities based on what he had heard. A cold, mechanical mind that was malfunctioning a bit was bad enough. A crazy, cold, mechanical mind was even worse.

Once back on the bridge, he settled in, waiting to see what the dark ships had done when they caught sight of Geary’s fleet coming at them from an unexpected direction.

“We should be seeing their reaction in less than a minute, Captain,” Lieutenant Yuon said.

“I don’t know why we’re so tense waiting to see,” Desjani grumbled to Geary. “We know they’re going to come around and charge toward an intercept with us.”

“The question is exactly how they’re going to do that,” he replied.

“If they’re mimicking your tactics, they’ll shift into three subformations,” she repeated.

His display lit up with alerts as the movements of the dark ships were finally seen. Nearly three and a half hours ago, the dark ships had pivoted about and begun accelerating toward an intercept with Geary’s fleet. As they did so, the dark ships had split from their one, massive arrangement, into a big central formation and two smaller formations on the wings of the first. The largest remained a rectangular box, while the two smaller were square boxes. “Oh, please,” Desjani scoffed. “They are dangling those small formations as obvious bait. Do they really think that you’ll fall for that?”

“I might have tried it,” Geary said. “Those side formations are very tempting.” The big central formation held twelve dark battleships, but each formation off to the side held only two battleships. “I want to hit them.”

“Would you, though? Would they expect Black Jack to do that?”

“Yes.” He reached toward the depiction of the dark ships on his display, using one finger to trace the formations the enemy warships were falling into as they accelerated toward Geary’s warships. “But the angle we’re going to intercept them at makes the small formation off to our port side the preferable target. Normally, I’d go for that.”

Desjani cocked a skeptical eye his way. “Going for the small formation to our starboard would mean cutting close across the path of their big formation.”

“Right. Which makes that a bad option.” He touched the main dark ship formation. “So, we’re going for this.”

She sat up, staring at him. “I know I said we should do something stupid, but I didn’t mean something that stupid.”

“We’re coming in a little higher than them, and with them slightly to our starboard,” Geary explained. With both formations heading for the fastest intercept, their relative positions wouldn’t change as the distance between them rapidly grew less. “Their artificial Geary will tell them that I will plan on hitting their port wing formation. They’ll plan on a last-moment shift in vector that will swing them toward where we’d be.”

He gestured on his display and the depiction of the dark ship main formation swung slightly over and up. “But if we actually aim for this point, where nothing is…” His finger rested above and to port of the enemy path.

“We’ll be able to clip the upper edge of their main formation and concentrate our fire on the two battleships there,” Desjani said, nodding. “But if they don’t do what you expect, the firing pass will be wasted. We won’t have anything within range. Neither will they, but they can afford to waste firing passes. We can’t.”

“I think it’s our best option,” Geary said.

“It’s definitely a good option.” She bent another questioning look on him. “It’s not stupid, either. Are we reserving the stupid options for later?”

“Yeah.” He rubbed his chin, gazing at his display. “I just have a feeling that we shouldn’t go stupid right away.”