David Deutch - The Fabric of Reality

The program in a virtual-reality generator embodies a general, predictive theory of the behaviour of the rendered environment. The other components deal with keeping track of what the user is doing and with the encoding and decoding of sensory data; these, as I have said, are relatively trivial functions. Thus if the environment is physically possible, rendering it is essentially equivalent to finding rules for predicting the outcome of every experiment that could be performed in that environment. Because of the way in which scientific knowledge is created, ever more accurate predictive rules can be discovered only through ever better explanatory theories. So accurately rendering a physically possible environment depends on understanding its physics.

The converse is also true: discovering the physics of an environment depends on creating a virtual-reality rendering of it. Normally one would say that scientific theories only describe and explain physical objects and processes, but do not render them. For example, an explanation of eclipses of the Sun can be printed in a book. A computer can be programmed with astronomical data and physical laws to predict an eclipse, and to print out a description of it. But rendering the eclipse in virtual reality would require both further programming and further hardware. However, those are already present in our brains! The words and numbers printed by the computer amount to ‘descriptions’ of an eclipse only because someone knows the meanings of those symbols. That is, the symbols evoke in the reader’s mind some sort of likeness of some predicted effect of the eclipse, against which the real appearance of that effect will be tested. Moreover, the ‘likeness’ that is evoked is interactive. One can observe an eclipse in many ways: with the naked eye, or by photography, or using various scientific instruments; from some positions on Earth one will see a total eclipse of the Sun, from other positions a partial eclipse, and from anywhere else no eclipse at all. In each case an observer will experience different images, any of which can be predicted by the theory. What the computer’s description evokes in a reader’s mind is not just a single image or sequence of images, but a general method of creating many different images, corresponding to the many ways in which the reader may contemplate making observations. In other words, it is a virtual-reality rendering. Thus, in a broad enough sense, taking into account the processes that must take place inside the scientist’s mind, science and the virtual-reality rendering of physically possible environments are two terms denoting the same activity.

Now, what about the rendering of environments that are not physically possible? On the face of it, there are two distinct types of virtual-reality rendering: a minority that depict physically possible environments, and a majority that depict physically impossible environments. But can this distinction survive closer examination? Consider a virtual-reality generator in the act of rendering a physically impossible environment. It might be a flight simulator, running a program that calculates the view from the cockpit of an aircraft that can fly faster than light. The flight simulator is rendering that environment. But in addition the flight simulator is itself the environment that the user is experiencing, in the sense that it is a physical object surrounding the user. Let us consider this environment. Clearly it is a physically possible environment. Is it a renderable environment? Of course. In fact it is exceptionally easy to render: one simply uses a second flight simulator of the same design, running the identical program. Under those circumstances the second flight simulator can be thought of as rendering either the physically impossible aircraft, or a physically possible environment, namely the first flight simulator. Similarly, the first flight simulator could be regarded as rendering a physically possible environment, namely the second flight simulator. If we assume that any virtual-reality generator that can in principle be built, can in principle be built again, then it follows that every virtual-reality generator, running any program in its repertoire, is rendering some physically possible environment. It may be rendering other things as well, including physically impossible environments, but in particular there is always some physically possible environment that it is rendering.

So, which physically impossible environments can be rendered in virtual reality? Precisely those that are not perceptibly different from physically possible environments. Therefore the connection between the physical world and the worlds that are renderable in virtual reality is far closer than it looks. We think of some virtual-reality renderings as depicting fact, and others as depicting fiction, but the fiction is always an interpretation in the mind of the beholder. There is no such thing as a virtual-reality environment that the user would be compelled to interpret as physically impossible.

We might choose to render an environment as predicted by some ‘laws of physics’ that are different from the true laws of physics. We may do this as an exercise, or for fun, or as an approximation because the true rendering is too difficult or expensive. If the laws we are using are as close as we can make them to real ones, given the constraints under which we are operating, we may call these renderings ‘applied mathematics’ or ‘computing’. If the rendered objects are very different from physically possible ones, we may call the rendering ‘pure mathematics’. If a physically impossible environment is rendered for fun, we call it a ‘video game’ or ‘computer art’. All these are interpretations. They may be useful interpretations, or even essential in explaining our motives in composing a particular rendering. But as far as the rendering itself goes there is always an alternative interpretation, namely that it accurately depicts some physically possible environment.

It is not customary to think of mathematics as being a form of virtual reality. We usually think of mathematics as being about abstract entities, such as numbers and sets, which do not affect the senses; and it might therefore seem that there can be no question of artificially rendering their effect on us. However, although mathematical entities do not affect the senses, the experience of doing mathematics is an external experience, no less than the experience of doing physics is. We make marks on pieces of paper and look at them, or we imagine looking at such marks — indeed, we cannot do mathematics without imagining abstract mathematical entities. But this means imagining an environment whose ‘physics’ embodies the complex and autonomous properties of those entities. For example, when we imagine the abstract concept of a line segment which has no thickness, we may imagine a line that is visible but imperceptibly wide. That much may, just about, be arranged in physical reality. But mathematically the line must continue to have no thickness when we view it under arbitrarily powerful magnification. That is not a property of any physical line, but it can easily be achieved in the virtual reality of our imagination.

Imagination is a straightforward form of virtual reality. What may not be so obvious is that our ‘direct’ experience of the world through our senses is virtual reality too. For our external experience is never direct; nor do we even experience the signals in our nerves directly — we would not know what to make of the streams of electrical crackles that they carry. What we experience directly is a virtual-reality rendering, conveniently generated for us by our unconscious minds from sensory data plus complex inborn and acquired theories (i.e. programs) about how to interpret them.

We realists take the view that reality is out there: objective, physical and independent of what we believe about it. But we never experience that reality directly. Every last scrap of our external experience is of virtual reality. And every last scrap of our knowledge — including our knowledge of the non-physical worlds of logic, mathematics and philosophy, and of imagination, fiction, art and fantasy — is encoded in the form of programs for the rendering of those worlds on our brain’s own virtual-reality generator.