Ray Kurzweil - How to Create a Mind - The Secret of Human Thought Revealed

In 1941 German scientist Konrad Zuse (1910–1995) created the Z-3 computer. It also read its program from a tape (in this case, coded on film) and also had no conditional branch instruction. Interestingly, Zuse had support from the German Aircraft Research Institute, which used the device to study wing flutter, but his proposal to the Nazi government for funding to replace his relays with vacuum tubes was turned down. The Nazis deemed computation as “not war important.” That perspective goes a long way, in my view, toward explaining the outcome of the war.

There is actually one genuine forerunner to von Neumann’s concept, and it comes from a full century earlier! English mathematician and inventor Charles Babbage’s (1791–1871) Analytical Engine, which he first described in 1837, did incorporate von Neumann’s ideas and featured a stored program via punched cards borrowed from the Jacquard loom. 4 Its random access memory included 1,000 words of 50 decimal digits each (the equivalent of about 21 kilobytes). Each instruction included an op code and an operand number, just like modern machine languages. It did include conditional branching and looping, so it was a true von Neumann machine. It was based entirely on mechanical gears and it appears that the Analytical Engine was beyond Babbage’s design and organizational skills. He built parts of it but it never ran. It is unclear whether the twentieth-century pioneers of the computer, including von Neumann, were aware of Babbage’s work.

Babbage’s computer did result in the creation of the field of software programming. English writer Ada Byron (1815–1852), Countess of Lovelace and the only legitimate child of the poet Lord Byron, was the world’s first computer programmer. She wrote programs for the Analytical Engine, which she needed to debug in her own mind (since the computer never worked), a practice well known to software engineers today as “table checking.” She translated an article by the Italian mathematician Luigi Menabrea on the Analytical Engine and added extensive notes of her own, writing that “the Analytical Engine weaves algebraic patterns, just as the Jacquard loom weaves flowers and leaves.” She went on to provide perhaps the first speculations on the feasibility of artificial intelligence, but concluded that the Analytical Engine has “no pretensions whatever to originate anything.”

Babbage’s conception is quite miraculous when you consider the era in which he lived and worked. However, by the mid-twentieth century, his ideas had been lost in the mists of time (although they were subsequently rediscovered). It was von Neumann who conceptualized and articulated the key principles of the computer as we know it today, and the world recognizes this by continuing to refer to the von Neumann machine as the principal model of computation. Keep in mind, though, that the von Neumann machine continually communicates data between its various units and within these units, so it could not be built without Shannon’s theorems and the methods he devised for transmitting and storing reliable digital information.

That brings us to the fourth important idea, which is to go beyond Ada Byron’s conclusion that a computer could not think creatively and find the key algorithms employed by the brain and then use these to turn a computer into a brain. Alan Turing introduced this goal in his 1950 paper “Computing Machinery and Intelligence,” which includes his now-famous Turing test for ascertaining whether or not an AI has achieved a human level of intelligence.

In 1956 von Neumann began preparing a series of lectures intended for the prestigious Silliman lecture series at Yale University. Due to the ravages of cancer, he never delivered these talks nor did he complete the manuscript from which they were to be given. This unfinished document nonetheless remains a brilliant and prophetic foreshadowing of what I regard as humanity’s most daunting and important project. It was published posthumously as The Computer and the Brain in 1958. It is fitting that the final work of one of the most brilliant mathematicians of the last century and one of the pioneers of the computer age was an examination of intelligence itself. This project was the earliest serious inquiry into the human brain from the perspective of a mathematician and computer scientist. Prior to von Neumann, the fields of computer science and neuroscience were two islands with no bridge between them.

Von Neumann starts his discussion by articulating the similarities and differences between the computer and the human brain. Given when he wrote this manuscript, it is remarkably accurate. He noted that the output of neurons was digital—an axon either fired or it didn’t. This was far from obvious at the time, in that the output could have been an analog signal. The processing in the dendrites leading into a neuron and in the soma neuron cell body, however, was analog, and he described its calculations as a weighted sum of inputs with a threshold. This model of how neurons work led to the field of connectionism, which built systems based on this neuron model in both hardware and software. (As I described in the previous chapter, the first such connectionist system was created by Frank Rosenblatt as a software program on an IBM 704 computer at Cornell in 1957, immediately after von Neumann’s draft lectures became available.) We now have more sophisticated models of how neurons combine inputs, but the essential idea of analog processing of dendrite inputs using neurotransmitter concentrations has remained valid.

Von Neumann applied the concept of the universality of computation to conclude that even though the architecture and building blocks appear to be radically different between brain and computer, we can nonetheless conclude that a von Neumann machine can simulate the processing in a brain. The converse does not hold, however, because the brain is not a von Neumann machine and does not have a stored program as such (albeit we can simulate a very simple Turing machine in our heads). Its algorithm or methods are implicit in its structure. Von Neumann correctly concludes that neurons can learn patterns from their inputs, which we have now established are coded in part in dendrite strengths. What was not known in von Neumann’s time is that learning also takes place through the creation and destruction of connections between neurons.

Von Neumann presciently notes that the speed of neural processing is extremely slow, on the order of a hundred calculations per second, but that the brain compensates for this through massive parallel processing—another unobvious and key insight. Von Neumann argued that each one of the brain’s 10 10neurons (a tally that itself was reasonably accurate; estimates today are between 10 10and 10 11) was processing at the same time. In fact, each of the connections (with an average of about 10 3to 10 4connections per neuron) is computing simultaneously.

Von Neumann’s estimates and his descriptions of neural processing are remarkable, given the primitive state of neuroscience at the time. One aspect of his work that I do disagree with, however, is his assessment of the brain’s memory capacity. He assumes that the brain remembers every input for its entire life. Von Neumann assumes an average life span of 60 years, or about 2 × 10 9seconds. With about 14 inputs to each neuron per second (which is actually low by at least three orders of magnitude) and with 10 10neurons, he arrives at an estimate of about 10 20bits for the brain’s memory capacity. The reality, as I have noted earlier, is that we remember only a very small fraction of our thoughts and experiences, and even these memories are not stored as bit patterns at a low level (such as a video image), but rather as sequences of higher-level patterns.

As von Neumann describes each mechanism in the brain, he shows how a modern computer could accomplish the same thing, despite their apparent differences. The brain’s analog mechanisms can be simulated through digital ones because digital computation can emulate analog values to any desired degree of precision (and the precision of analog information in the brain is quite low). The brain’s massive parallelism can be simulated as well, given the significant speed advantage of computers in serial computation (an advantage that has vastly expanded over time). In addition, we can also use parallel processing in computers by using parallel von Neumann machines—which is exactly how supercomputers work today.