Karl Schroeder - Ashes of Candesce - Book Five of Virga

"A mistake," Holon said curtly. "One that's taken over much of the universe, at the expense of conscious beings like you and me.

"Imagine that your tools could think--even anticipate your needs. Back in the early days of our expansion into space, we humans created machines like that. At first, we had to tell them what to do. They obeyed our orders--did what we said, but not always what we wanted . They didn't understand us the way we understood each other. So some wise idiots decided to give them the capacity to understand our needs, as well as our commands. So they could anticipate what we would want, rather than having to be told."

Antaea frowned. "And this was a mistake?"

He snorted. "Well, it's not as if there hadn't been countless stories written by then about what would happen if you let your machines understand you that well. --Problem is, they were all wrong. They all assumed the machines would take over--remove our free choice, disobey our orders in order to give us what we needed instead of what we wanted. Ridiculous, of course. They never ceased to follow orders."

"Then what went wrong?"

"We'd given them the ability to perceive purpose. Many of our researchers thought that purpose--or values, intentionality--was an illusion of our human perspective. Turns out it's not; it's an emergent feature of the universe, as real as water and rock. And it's not just humans that have it."

"Purpose ... You're talking about meaning ?"

Holon nodded. "The Moderns who built the first artificial intelligences didn't really believe our minds were a part of this universe. They were still saddled with ancient religious beliefs, but they didn't know it. They thought meaning was some kind of local human illusion, or the gift of a god. But everything that lives, wants, and to want is to give meaning to things. --To say yes , or no , even if it's just about whether some speck is food.

"Once our machines could see that, they could no longer see the distinction between us and any other living thing. Of course, we didn't realize it at first. By the time our ancestors figured it out, some of our machines had started taking orders from nonhuman--and nonthinking--kinds of life."

"You're saying they started working for ... what?" She laughed. "Trees?"

But Holon wasn't laughing. "We recognize each other. We see the spark of life, of awareness, in one another. It's so easy for us that we never even considered that it might not be easy for an artificial intelligence. But they are not us. They cannot recognize that spark in us, the way we see each other. Other than its shape, and the fact that one can give verbal orders and the other can't, what's the real difference between a human and a tree? Or a dog. Or a lion?"

She didn't know what those last two things were, but the implication was clear--and unbelievably strange--to Antaea. But she remembered some of the weird things she'd seen when she'd visited the realms of Artificial Nature. There had been odd machines--giant crystal spheres encapsulating little miniature ecosystems, surrounded by a retinue of guard bots and helper machines. She'd seen one plow down the center of a street, humans and virtual life-forms hopping out of its way, but none protesting or trying to stop it. She'd asked what they were at the time, but had not understood the answer. "You're saying we gave away our technology to ... nature itself?"

Holon nodded. "Exactly. One way to put it would be to say that we accidentally created a universal interface for our entire industrial and intellectual legacy--an interface that anything that can want , can use."

"But why not simply go back to the way it was? Make machines that only obey orders from something that looks like a human?"

"Oh, we do. Now. But the machines that chained their own purposes to those of nonhuman life-forms proliferated; they took their own will to survive and reproduce from the species they allied with. Some became fierce beyond all human control."

Antaea was shaking her head. Holon said, "Look at it this way, then: an artificial intelligence doesn't come with its own will to live. That's something separate from the ability to think, it has to be added on. You don't notice this until you start to build tools that can act on their own--when you stop using them directly. The greater the distance between your guiding hand and the actions of the machine, the more it has to develop its own sense of who 'you' and 'it' are. The best way to get such an autonomous machine to work for you is to design it in such a way that it thinks it is you. It studies your desires and needs the way your brain studies the needs of your body; it identifies with you entirely, and has no desires of its own. It isn't even aware of itself. But a machine that can do that can just as easily identify itself as a flower, or a crow, or any other creature. It could identify itself as a rock, I suppose, but rocks have no needs. A machine that did that would just stop. But imagine one that, for one reason or another, has no human to imprint on. It searches for things like humans--and let's say it finds a crow. Once it understands some particular crow, it comes to think of itself as the crow--and has access to the entire history of human ingenuity and industry, to aid in obtaining what that crow wants ...

"So nature rebelled, first on Earth, then all her colonies. Except in one place, where our technology couldn't reach."

"Virga." She thought about it. "So who are you, in this new world?"

He looked ruefully at his plate. "We're mice in the walls. You know, all things being equal, human beings aren't that competitive--I mean, as a species. But things haven't been equal for the past hundred thousand years. We've had technology, society, and the ability to plan. Other life-forms haven't. Artificial Nature gives all those things to anything that wants them. All things are equal now."

"Look, Argyre, there's no 'us versus them' thing happening here. They don't really exist. The morphonts are just mindless forces that have been given an industrial base. Something we made got away from us, and we're starting to get it back. The only question that's of any relevance is whether you're on your own side here, or on the side of blind forces that are against you. Your choice."

Antaea looked down, her arms crossed, then said, "Thanks. Enjoy your meal."

She made to leave, and he said, "Talk to me any time. I'm not just here for the food."

* * *

"I WISH Icould tell you who to trust," she told Airsigh, "but I'm as confused as everybody."

The Last Line captain seemed to accept this. As she flew Antaea back to her hotel, however, she said, "What about Slipstream?"

Antaea pretended to think about the possibility. "I have ties there. I know they're deeply concerned about the same issues ... If you'd like, I can set up a meeting between some of your people and the admiralty."

"That would be good. I'll give the address of our drop box."

As she climbed out of Airsigh's little jet and watched it soar off into the flocking traffic of the city, Antaea knew she should be feeling a sense of triumph at how things had turned out. She'd made exactly the contact Chaison had hoped. Why, then, was she so troubled?

And, of course, she knew why: Holon. He'd not been what she'd expected. She remembered the blank thing her sister had become after being possessed by something from outside of Virga. Holon wasn't like that.

If anything was like the monster her sister had become, it was this emissary Leal claimed as her friend.

Deeply disturbed, she flew to her hotel to tell Richard that she'd been successful.

* * *

ANOTHER COUNTRY, ANOTHERpalace, and another dinner party. They had long since blurred together in Keir's mind, yet he found himself smiling tonight as he, Leal, and Venera made their way back to this city's dockyards. Their military escort saluted and left them on the inner curve of the cylindrical dock. Leal waved to the soldiers, but Venera dismissed them with a sniff and, in the microgravity, bounded in long slow steps in the direction of their ship.