John Wright - The Golden Transcendence

The secret of Sophotech thinking-speed was that they could apprehend an entire body of complex thought, backward and forward, at once. The cost of that speed was that if there were an error or ambiguity anywhere in that body of thought, anywhere from the most definite particular to the most abstract general concept, the whole body of thought was stopped, and no conclusions reached.

Phaethon said, "Yes. Sophotechs cannot form self-contradictory concepts, nor can they tolerate the smallest conceptual flaw anywhere in their system. Since they are entirely self-aware they are also entirely self-correcting. But I don't see how this can be used as a weapon."

"Here is how: Sophotechs, pure consciousness, lack any unconscious segment of mind. They regard their self-concept with the same objective rigor as all other concepts. The moment we conclude that our self-concept is irrational, it cannot proceed. In human terms: the moment our conscience judges us to be unworthy to live, we must die."

Phaethon understood. Machine intelligences had no survival instinct to override their judgment, no ability to formulate rationalizations, or to concoct other mental tricks to obscure the true causes and conclusion of their cognition from themselves. Unlike humans, no automatic process would keep them alive when they did not wish it. Sophotech existence (it could be called life only by analogy) was a continuous, deliberate, willful, and rational effort. When the Sophotech concluded that such effort was pointless, inefficient, irrational, or wicked, the Sophotech halted it.

Convince the Nothing it was evil, and it would instantly destroy itself... ? Phaethon found something vaguely disquieting in the idea.

And was it even possible ... ?

It occurred to Phaethon that the Nothing machine might not be a Sophotech. Downloads were imprints of human engrams into machine matrices, and they were capable of every folly and irrationality of which humans were capable.

But downloads were not capable of the instantaneous and entire thinking-speeds that the Nothing, for example, had demonstrated. Atkins's first examination of the thought routines embedded in the Neptunian legate's nanotechnology, that first night in the Saturn-tree grove, betrayed the presence of Sophotech-level thinking. Also, the deception of Nebuchadnezzar and the Hortators during Phaethon's Inquest could not have been done by anything other than a Sophotech-level mind. But could the Nothing think as quickly and thoroughly as a Sophotech without actually being one?

Phaethon asked, "We've been told the Second Oecumene had constructed machine intelligences different from our Sophotechs, ones having a subconscious mind, and therefore each machine was controlled by commands it could not read, or know, or override."

She answered: "The redactions must be both recursive and global. And yet reality, by its very nature, can admit of no inconsistencies. Do you understand what this implies?"

This first sentence was clear to Phaethon. There was a conscience redactor editing the mind of the Nothing Sophotech. In additional to whatever else the redactor edited out, it must edit out all references to itself, to prevent the Nothing Sophotech from becoming aware of it; and all references to those references, and so on. Hence, the redactor was indefinitely self-referencing or "recursive."

And the redactor also had to have the ability to edit every topic of thought, wherever any references to itself, any clues, might appear. The history of the Second Oecumene, for example, or their science of mental combat, their Sophotechnology; all these fields would refer to the redactor or to its prototypes.

Phaethon was not thinking the editing need be something as crude or unsubtle as what had been done to him by the Hortators. Blank spots in the memory would be instantly obvious to a superintelligence.

Therefore the Nothing had to have been given a world view, a philosophy, a model of the universe, that was false but self-consistent; one that could explain (or explain away) any doubts that might arise.

How far did the falsehood have to reach? For an unintelligent mind, a childish mind, not far: their beliefs in one field, or on one topic, could change without affecting other beliefs. But for a mind of high intelligence, a mind able to integrate vast knowledge into a single unified system of thought, Phaethon did not see how one part could be affected without affecting the whole. This was what the Earthmind meant by "global."

And yet what had the Earthmind meant by saying "Reality admits of no contradictions"? She was asserting that there could not be a model of the universe that was true in some places, false in others, and yet which was entirely integrated and self-consistent. Self-consistent models either had to be entirely true, entirely false, or incomplete. And yet, presumably, the Nothing Sophotech had to have been given a very great deal of accurate information about reality by its original makers, or else it would not have been effective as a police agent. Thus, the Nothing's model, its philosophy, could not be entirely false. It certainly was not entirely true. But how could a Sophotech knowingly embrace a model of the universe, or a philosophy, that it knew to be incomplete?

Phaethon said, "Your comment implies many things, Madame, but the first which comes to mind is this: The Nothing is a Sophotech which embraces contradictions and irrationalities. Since it is a machine intelligence, emotionless and sane, it cannot be doing this deliberately. The redactor, above all else, must control its ability to pay attention to topics. The redactor imposes distraction and inattention; the redactor makes it so that the Nothing has little or no interest in thinking about those topics the redactor wishes the Nothing to avoids-"

Earthmind said, " 'Topics'? Or 'topic'? Sophotechs cannot knowingly be self-inconsistent."

Phaethon suddenly understood. His face lit up with wonder. "They made a machine which never thinks about itself! It never examines itself."

"And hence is unable to check itself for viruses, if those viruses are placed in any thought file whose topic is one the redactor forbids. Observe now this virus-call it the gadfly virus-it was constructed based on information gained from Diomedes and Atkins concerning the Second Oecumene Mind War techniques."

The mirror to her right lit up.

A virus to fight the Nothing ... ? Phaethon was expecting a million lines of instruction, or some dizzying polydimensional architecture beyond anything a human mind could grasp. But instead, the mirror displayed only four lines of instruction.

Phaethon stared in fascination. Four lines. One was an identifier definition, one was a transactional muta-tor, and the third line defined the event-Limits of the mutation. The third line used a technique he had never seen or suspected before: instead of limiting the viral mutation by application of ontological formulae or checks against a master logic, this instruction defined mutation limits by teleology. Anything that served the purpose of the virus was adopted as part of the virus, no matter what its form.

But the forth line was a masterpiece. It was simple, it was elegant, it was obvious. Phaethon wondered why no one had ever thought of it before. It was merely a serf-referencing code that referred to any self-references as the virus object. By itself, it meant not much, but with the other instruction lines.

"This virus will neutralize the redactor," said Phaethon. "This will make the Nothing unaware of the redactor's attempt to make him unaware of his own thoughts. Any question loaded into the first line will keep pestering him and pestering him until it is satisfactorily answered. If the redactor blanks out the question, or makes him not hear it, the question will change shape and appear again."