Ray Kurzweil - How to Create a Mind - The Secret of Human Thought Revealed

Von Neumann concludes that the brain’s methods cannot involve lengthy sequential algorithms, when one considers how quickly humans are able to make decisions combined with the very slow computational speed of neurons. When a third baseman fields a ball and decides to throw to first rather than to second base, he makes this decision in a fraction of a second, which is only enough time for each neuron to go through a handful of cycles. Von Neumann concludes correctly that the brain’s remarkable powers come from all its 100 billion neurons being able to process information simultaneously. As I have noted, the visual cortex makes sophisticated visual judgments in only three or four neural cycles.

There is considerable plasticity in the brain, which enables us to learn. But there is far greater plasticity in a computer, which can completely restructure its methods by changing its software. Thus, in that respect, a computer will be able to emulate the brain, but the converse is not the case.

When von Neumann compared the capacity of the brain’s massively parallel organization to the (few) computers of his time, it was clear that the brain had far greater memory and speed. By now the first supercomputer to achieve specifications matching some of the more conservative estimates of the speed required to functionally simulate the human brain (about 10 16operations per second) has been built. 5 (I estimate that this level of computation will cost $1,000 by the early 2020s.) With regard to memory we are even closer. Even though it was remarkably early in the history of the computer when his manuscript was written, von Neumann nonetheless had confidence that both the hardware and software of human intelligence would ultimately fall into place, which was his motivation for having prepared these lectures.

Von Neumann was deeply aware of the increasing pace of progress and its profound implications for humanity’s future. A year after his death in 1957, fellow mathematician Stan Ulam quoted him as having said in the early 1950s that “the ever accelerating progress of technology and changes in the mode of human life give the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue.” This is the first known use of the word “singularity” in the context of human technological history.

Von Neumann’s fundamental insight was that there is an essential equivalence between a computer and the brain. Note that the emotional intelligence of a biological human is part of its intelligence. If von Neumann’s insight is correct, and if one accepts my own leap of faith that a nonbiological entity that convincingly re-creates the intelligence (emotional and otherwise) of a biological human is conscious (see the next chapter), then one would have to conclude that there is an essential equivalence between a computer— with the right software —and a (conscious) mind. So is von Neumann correct?

Most computers today are entirely digital, whereas the human brain combines digital and analog methods. But analog methods are easily and routinely re-created by digital ones to any desired level of accuracy. American computer scientist Carver Mead (born in 1934) has shown that we can directly emulate the brain’s analog methods in silicon, which he has demonstrated with what he calls “neuromorphic” chips. 6 Mead has demonstrated how this approach can be thousands of times more efficient than digitally emulating analog methods. As we codify the massively repeated neocortical algorithm, it will make sense to use Mead’s approach. The IBM Cognitive Computing Group, led by Dharmendra Modha, has introduced chips that emulate neurons and their connections, including the ability to form new connections. 7 Called “SyNAPSE,” one of the chips provides a direct simulation of 256 neurons with about a quarter million synaptic connections. The goal of the project is to create a simulated neocortex with 10 billion neurons and 100 trillion connections—close to a human brain—that uses only one kilowatt of power.

As von Neumann described over a half century ago, the brain is extremely slow but massively parallel. Today’s digital circuits are at least 10 million times faster than the brain’s electrochemical switches. Conversely, all 300 million of the brain’s neocortical pattern recognizers process simultaneously, and all quadrillion of its interneuronal connections are potentially computing at the same time. The key issue for providing the requisite hardware to successfully model a human brain, though, is the overall memory and computational throughput required. We do not need to directly copy the brain’s architecture, which would be a very inefficient and inflexible approach.

Let’s estimate what those hardware requirements are. Many projects have attempted to emulate the type of hierarchical learning and pattern recognition that takes place in the neocortical hierarchy, including my own work with hierarchical hidden Markov models. A conservative estimate from my own experience is that emulating one cycle in a single pattern recognizer in the biological brain’s neocortex would require about 3,000 calculations. Most simulations run at a fraction of this estimate. With the brain running at about 10 2(100) cycles per second, that comes to 3 × 10 5(300,000) calculations per second per pattern recognizer. Using my estimate of 3 × 10 8(300 million) pattern recognizers, we get about 10 14(100 trillion) calculations per second, a figure that is consistent with my estimate in The Singularity Is Near . In that book I projected that to functionally simulate the brain would require between 10 14and 10 16calculations per second (cps) and used 10 16cps to be conservative. AI expert Hans Moravec’s estimate, based on extrapolating the computational requirement of the early (initial) visual processing across the entire brain, is 10 14cps, which matches my own assessment here.

Routine desktop machines can reach 10 10cps, although this level of performance can be significantly amplified by using cloud resources. The fastest supercomputer, Japan’s K Computer, has already reached 10 16cps. 8 Given that the algorithm of the neocortex is massively repeated, the approach of using neuromorphic chips such as the IBM SyNAPSE chips mentioned above is also promising.

In terms of memory requirement, we need about 30 bits (about four bytes) for one connection to address one of 300 million other pattern recognizers. If we estimate an average of eight inputs to each pattern recognizer, that comes to 32 bytes per recognizer. If we add a one-byte weight for each input, that brings us to 40 bytes. Add another 32 bytes for downward connections, and we are at 72 bytes. Note that the branching-up-and-down figure will often be much higher than eight, though these very large branching trees are shared by many recognizers. For example, there may be hundreds of recognizers involved in recognizing the letter “p.” These will feed up into thousands of such recognizers at this next higher level that deal with words and phrases that include “p.” However, each “p” recognizer does not repeat the tree of connections that feeds up to all of the words and phrases that include “p”—they all share one such tree of connections. The same is true of downward connections: A recognizer that is responsible for the word “APPLE” will tell all of the thousands of “E” recognizers at a level below it that an “E” is expected if it has already seen “A,” “P,” “P,” and “L.” That tree of connections is not repeated for each word or phrase recognizer that wants to inform the next lower level that an “E” is expected. Again, they are shared. For this reason, an overall estimate of eight up and eight down on average per pattern recognizer is reasonable. Even if we increase this particular estimate, it does not significantly change the order of magnitude of the resulting estimate.