David Deutch - The Fabric of Reality

If an image generator is playing a recording taken from life, its accuracy may be defined as the closeness of the rendered images to the ones that a person in the original situation would have perceived. More generally, if the generator is rendering artificially designed images, such as a cartoon, or music played from a written composition, the accuracy is the closeness of the rendered images to the intended ones. By ‘closeness’ we mean closeness as perceived by the user. If the rendering is so close as to be indistinguishable by the user from what is intended, then we can call it perfectly accurate. (So a rendering that is perfectly accurate for one user may contain inaccuracies that are perceptible to a user with sharper senses, or with additional senses.)

A universal image generator does not of course contain recordings of all possible images. What makes it universal is that, given a recording of any possible image, it can evoke the corresponding sensation in the user. With a universal auditory sensation generator — the ultimate hi-fi system — the recording might be given in the form of a compact disc. To accommodate auditory sensations that last longer than the disc’s storage capacity allows, we must incorporate a mechanism that can feed any number of discs consecutively into the machine. The same proviso holds for all other universal image generators, for strictly speaking an image generator is not universal unless it includes a mechanism for playing recordings of unlimited duration. Furthermore, when the machine has been playing for a long time it will require maintenance, otherwise the images it generates will become degraded or may cease altogether. These and similar considerations are all connected with the fact that considering a single physical object in isolation from the rest of the universe is always an approximation. A universal image generator is universal only in a certain external context, in which it is assumed to be provided with such things as an energy supply, a cooling mechanism and periodic maintenance. That a machine has such external needs does not disqualify it from being regarded as a ‘single, universal machine’ provided that the laws of physics do not forbid these needs from being met, and provided that meeting those needs does not necessitate changing the machine’s design.

Now, as I have said, image generation is only one component of virtual reality: there is the all-important interactive element as well. A virtual-reality generator can be thought of as an image generator whose images are not wholly specified in advance but depend partly on what the user chooses to do. It does not play its user a predetermined sequence of images, as a movie or a feelie would. It composes the images as it goes along, taking into account a continuous stream of information about what the user is doing. Present-day virtual-reality generators, for instance, keep track of the position of the user’s head, using motion sensors as shown in Figure 5.1. Ultimately they will have to keep track of everything the user does that could affect the subjective appearance of the emulated environment. The environment may include the user’s own body: since the body is external to the mind, the specification of a virtual-reality environment may legitimately include the requirement that the user’s body should seem to have been replaced by a new one with specified properties.

The human mind affects the body and the outside world by emitting nerve impulses. Therefore a virtual-reality generator can in principle obtain all the information it needs about what the user is doing by intercepting the nerve signals coming from the user’s brain. Those signals, which would have gone to the user’s body, can instead be transmitted to a computer and decoded to determine exactly how the user’s body would have moved. The signals sent back to the brain by the computer can be the same as those that would have been sent by the body if it were in the specified environment. If the specification called for it, the simulated body could also react differently from the real one, for example to enable it to survive in simulations of environments that would kill a real human body, or to simulate malfunctions of the body.

I had better admit here that it is probably too great an idealization to say that the human mind interacts with the outside world only by emitting and receiving nerve impulses. There are chemical messages passing in both directions as well. I am assuming that in principle those messages could also be intercepted and replaced at some point between the brain and the rest of the body. Thus the user would lie motionless, connected to the computer, but having the experience of interacting fully with a simulated world — in effect, living there. Figure 5.2 illustrates what I am envisaging. Incidentally, though such technology lies well in the future, the idea for it is much older than the theory of computation itself. In the early seventeenth century Descartes was already considering the philosophical implications of a sense-manipulating ‘demon’ that was essentially a virtual-reality generator of the type shown in Figure 5.2, with a supernatural mind replacing the computer.

From the foregoing discussion it seems that any virtual-reality generator must have at least three principal components:

a set of sensors (which may be nerve-impulse detectors) to detect what the user is doing,

a set of image generators (which may be nerve-stimulation devices), and

a computer in control.

My account so far has concentrated on the first two of these, the sensors and the image generators. That is because, at the present primitive state of the technology, virtual-reality research is still preoccupied with image generation. But when we look beyond transient technological limitations, we see that image generators merely provide the interface — the ‘connecting cable’ — between the user and the true virtual-reality generator, which is the computer. For it is entirely within the computer that the specified environment is simulated. It is the computer that provides the complex and autonomous ‘kicking back’ that justifies the word ‘reality’ in ‘virtual reality’. The connecting cable contributes nothing to the user’s perceived environment, being from the user’s point of view ‘transparent’, just as we naturally do not perceive our own nerves as being part of our environment. Thus virtual-reality generators of the future would be better described as having only one principal component, a computer, together with some trivial peripheral devices.

FIGURE 5.2. Virtual reality as it might be implemented in the future.

I do not want to understate the practical problems involved in intercepting all the nerve signals passing into and out of the human brain, and in cracking the various codes involved. But this is a finite set of problems that we shall have to solve once only. After that, the focus of virtual-reality technology will shift once and for all to the computer, to the problem of programming it to render various environments. What environments we shall be able to render will no longer depend on what sensors and image generators we can build, but on what environments we can specify. ‘Specifying’ an environment will mean supplying a program for the computer, which is the heart of the virtual-reality generator.

Because of the interactive nature of virtual reality, the concept of an accurate rendering is not as straightforward for virtual reality as it is for image generation. As I have said, the accuracy of an image generator is a measure of the closeness of the rendered images to the intended ones. But in virtual reality there are usually no particular images intended: what is intended is a certain environment for the user to experience. Specifying a virtual-reality environment does not mean specifying what the user will experience, but rather specifying how the environment would respond to each of the user’s possible actions. For example, in a simulated tennis game one may specify in advance the appearance of the court, the weather, the demeanour of the audience and how well the opponent should play. But one does not specify how the game will go: that depends on the stream of decisions the user makes during the game. Each set of decisions will result in different responses from the simulated environment, and therefore in a different tennis game.