Bruce Mesquita - The Predictioneer's Game

Today, the governments of the United States, China, Russia, Japan, and the Republic of Korea try to work out how to bring Kim Jong Il’s rogue state into the mainstream of international affairs. For years he was threatened, he was coaxed, he was urged to be better, but he also was rewarded by Bill Clinton, at Jimmy Carter’s behest, in the hope that North Korea’s behavior would improve. The rewards Kim’s regime got—in the form of foreign aid, technology transfers, and food relief—came with little or no genuine commitment on Kim’s part, just empty promises. More recently, the six-nation talks seem to have made progress. The task of diminishing his threat appears to be moving forward, although in fits and starts. Finding a path to resolving his nuclear threat can be, and perhaps has been, advanced by accurate predictions and engineering aimed at getting Kim to put his bombs away.

In early 2004 the Department of Defense hired me as a consultant to investigate alternative scenarios for trying to get North Korea to behave better on the nuclear front. I can only sketch the solutions I came up with here, but even so, we can see how to think about such issues. I will focus attention on the scenario that held out the greatest promise for solving the problem. This scenario looks at trade-offs between U.S. political and economic concessions in exchange for North Korean concessions on the nuclear front. Before delving into details, however, let me be perfectly clear that I am taking and seek no credit for whatever progress may have been made. Policy consultants almost never know whether they are being listened to. They almost never know what other sources of advice are being heard or what those sources are saying. All I can do is report what my analysis showed and relate it to what eventually happened.

What kind of information is needed to make reliable predictions and tactical or even strategic recommendations? First, of course, it is essential to define the questions for which we want answers. A question like “How can we get Kim Jong Il to behave better?” is too vague. We need to define the objective more precisely, and we need to know the range of choices that Kim and his government can undertake. In this case, the choices included the possibility that Kim and his regime would refuse to negotiate about nuclear weapons at all; would negotiate but then cheat on any agreement as soon as doing so became advantageous (Kim’s preferred approach); would slowly reduce the extent of North Korea’s nuclear program in exchange for different levels of U.S. economic and security concessions; would eliminate the program conditionally (with various gradations of conditions having been specified); or would eliminate the nuclear program unconditionally (the most preferred outcome from the perspective of the American president and his foreign policy team).

Next we want to know what background conditions should be imposed on the question. For example, we might ask which of the above policies Kim Jong Il could be induced to adopt if the United States publicly targeted nuclear-tipped missiles at North Korea, or if the United States guaranteed North Korea’s security within its borders, or a host of other possibilities. Each condition defines a scenario so that we can compare what is likely to happen if the United States (or some other government) takes this or that action. This way, we start to answer “what if” questions. Once the issue, the options, and the scenarios are defined, then only a very few facts and a bit of logic are needed to identify solutions.

First, the facts. In my experience, all that is necessary to make a reliable prediction is to:

Identify every individual or group with a meaningful interest in trying to influence the outcome. Don’t just pay attention to the final decision makers.

Estimate as accurately as possible with available information what policy each of the players identified in point 1 is advocating when they talk in private to each other—that is, what do they say they want.

Approximate how big an issue this is for each of the players—that is, how salient is it to them. Are they so concerned that they would drop whatever they’re doing to address this problem when it comes up, or are they likely to want to postpone discussions while they deal with more pressing matters?

Relative to all of the other players, how influential can each player be in persuading others to change their position on the issue?

That’s all you need to know. That’s all? you might ask. What about history? What about culture? What about personality traits? What about almost everything else that most people think is important to know? Knowing all of those things would be great, and I will say more about them in a moment, but none of that information is crucial to making correct forecasts or to engineering policy change. Sure, it helps. It is generally better to know more than less. Still, anyone who does not put together information on the four factors I listed above is unlikely to assess a situation correctly.

What’s interesting about the four pieces of information that I contend are crucial is that while they’re not the kind of stuff that can be easily looked up in a book, it is possible to tease such information out of articles in The Economist, US News & World Report, Time, Newsweek, The Financial Times, The New York Times, The Wall Street Journal , and from Internet stories and other news outlets. Understanding the availability of such information, and having the confidence to employ it, is a big part of predicting and engineering outcomes. Admittedly, it is a lot of work going through so many news sources, and for problems with a short fuse, that approach can take too long. Luckily, there is a more efficient way to get the information—ask the experts. It’s that simple.

Experts have invested years in learning a place’s culture, language, and history. They follow the intimate political details that go on in the area they study. If anyone knows who will try to shape decisions, how influential those people can be, where they stand, and how much they care about an issue, it is the experts. Come to think of it, isn’t knowing this information what it means to be an expert?

About now you might wonder, if the experts know the information needed to make predictions, what do we need a predictioneer for? Here is where specialization of skills is really important. It’s important to remember that experts alone do not do nearly as well at anticipating developments as do experts combined with a good model of how people think. A declassified CIA study reports that my forecasting model has hit the bull’s-eye about twice as often as the government’s experts who provided me with data. 1I certainly don’t know more than they do about the countries or problems they study. In fact, I often know no more than what they tell me. But they’re not experts on how people make choices, because that, after all, is not the focus of their knowledge.

With all of the information we collect in order to predict and shape outcomes, computer modeling is necessary both to organize the data and to run simulations of negotiations or exchanges. Think of these simulations as a game of chess in many dimensions, in which the computer calculates everyone’s expected actions, taking anticipated responses by everyone else into account. The computer has a tremendous advantage over experts, analysts, or the smartest decision makers when it comes to playing such a complicated game. Computers don’t get tired; they don’t get bored; they don’t need coffee breaks or much sleep; and they have fabulous memories. They are content to crunch as much information as we shovel into them.

Consider the computer’s advantage. Suppose we were examining the North Korean nuclear problem in 2004, as I was, and suppose we simplified it (which I didn’t) to consider just five players: George W. Bush, Kim Jong Il, Russia’s Vladimir Putin, China’s Hu Jintao, and South Korea’s Roh Moo Hyun (ignoring Japan for the moment). How many conversations among the parties to the talks might each of those five decision makers want to know about?