Bruce Mesquita - The Predictioneer's Game

George Bush certainly would want to keep track of what he said to each of the other four and what each of them said to him. They would all want to keep tabs as well on any proposals they made and any they heard from others. That’s twenty exchanges of views right there. Certainly that is not all that any of them would want to know. Bush would be interested to know, or at least try to figure out, what Kim Jong Il might be saying to Putin, to Hu Jintao, and to Roh, and each of them would want to know about the conversations they were not directly involved in too. That’s another sixty possible discussions. And Bush might even want to know what, for example, Putin thought Kim was saying to Hu Jintao and to Roh Moo Hyun, not to mention what he thought Roh said to Hu and to Kim, and so on.

All in all, taking all the layers of information being traded back and forth among just these five decision makers, there are 120 possible exchanges or imagined exchanges (that is, 5 factorial, or 5×4×3×2= 120) to know about. Keeping track of those 120 possible offers and counteroffers that might be on the table is essential in sorting out what is best to do at any moment in a negotiation. Those 120 possible exchanges of points of view and beliefs about such exchanges are what can happen in a single round of bargaining with just five stakeholders. It might be surprising to know that a smart person can keep that amount of information pretty straight in his or her head. Keeping the information straight, however, becomes an acute problem as the number of interested parties rises.

Just adding Japan’s prime minister—the talks are, after all, six-party talks, not five-party talks—to the mix inflates the number of important bits of information six times from 120 to 720. Moving up just to ten players, the number of useful pieces of information rises astonishingly to over 3.6 million! No one—not Newton, not Einstein, not von Neumann—can keep that much information straight in his head; but of course the tireless computer can.

Alas, the computer’s great memory and excellent work habits come at a price. There is a vital gap between how experts or newspaper articles express facts and how computers ingest and digest them. Like the rest of us, experts communicate in sentences. Models talk in numbers. So part of my job is to turn sentences into numbers so that the computer can crunch away. Numbers have big advantages over words—and not just for computers. Most importantly, numbers are clear; words are vague. It’s essential to turn information into numerical values, and in fact it’s not especially hard to do.

To get a sense of how readily experts know the information needed to make reliable predictions, and to see how easily the information can be turned into numbers, try an experiment. Interview someone you think of as really knowing about your friends and family, including perhaps yourself. Pick an issue that is important to your family or friends. It doesn’t have to be about world affairs; it could be about where to have dinner, or what movie to see, or whatever else leads to disagreements. The easiest sort of issue to do as a first try is what I call a beauty contest. Say you and some friends are trying to choose between two movies. Anyone who really truly wants to see The Sound of Music (or fill in whatever first-run movie might grab your fancy) gets a value of 100, and anyone really committed to seeing A Clockwork Orange (another great old film) gets a 0. Then you should be able to rate how strongly each friend leans toward one movie or the other. Any who are truly indifferent get a 50; anyone leaning slightly toward A Clockwork Orange might be close to 50, say at 40 or 45, and so on. You or the “expert” you are interviewing need to calibrate their strength of feeling as accurately as possible. This way, movie preferences are turned into one value for each chooser—that is, each family member and/or friend involved in shaping the decision. The process is exactly the same, although the choices may be more complex, whether deciding what movie to see or addressing North Korea’s nuclear choices. Sure, the stakes differ, but once the essential facts are extracted, the process of turning stated objectives into a predicted outcome is the same.

Now estimate how eager each friend or family member—each player—is to weigh in on the decision. If you think a family member will drop what he or she is doing to discuss the movie to see, rate that person’s “salience” (variable 3 on my list) close to 100 (no one is ever really at 100). The less focused you think someone is on the movie choice, the lower the salience score. If a family member is the sort who would say, “Look, I’ll go to whatever movie you choose, but really I don’t have time to get involved in picking which one,” that’s somewhere around a 10. On the other hand, if you think a friend will say, “I’m busy right now but call me back in ten minutes,” that’s pretty high salience. “Call me back in an hour” is lower, and “Call me back next week” is much lower. With a bit of effort, it shouldn’t be that hard to calibrate how important the movie decision is to each person compared to the other decisions they have to make (not compared to each other, mind you, but compared to other things they need to do or deal with).

Finally, figure out who you think has the most influence among your friends or family members if you assume that everyone thinks the choice is equally important. Give the person credited as being most persuasive a score of 100 and rate everybody else relative to that. So if Harry is 100 and Jane is 60 and John is 40 in potential clout, and Jane and John want to see The Sound of Music and Harry wants to see A Clockwork Orange , then that means that John and Jane together just offset Harry’s ability to persuade if they all care equally intensely about choosing a movie. If Jane were 60 and John were 70, then, all else being equal, they could persuade Harry to moderate his view and give more consideration to the movie Jane and John prefer. Of course, if someone else supported Harry’s choice, that might create a strong enough coalition to defeat John and Jane. The dynamics get complicated, but the basic idea should be straightforward.

It’s important to note that most of us make these assessments of interests in any situation. We just do such calculations naturally with relative judgments of where people stand on a given issue. What I’ve sketched above is simply a formalization of that natural process—which becomes all the more needed the more complicated the problems in question become.

By now you are probably thinking, Sure, people can fill in numbers to the questions, but it’s just guesswork. Ask two experts the same question and you’ll get two different answers. Guess what—that’s not true. If it were, then there is barely any chance that a model like mine could achieve any consistent accuracy. There would be too much luck involved. In fact, the CIA has checked out the risk that different experts give greatly different answers leading to greatly different predictions. They found little variation in the predictive results from the sort of modeling I do, even when the people asked had dramatically different access to information. Academic experts, for instance, generally do not know the classified information that intelligence analysts have access to. Yet both groups tend to provide data so similar, wherever it’s from, that the results hardly change when moving across these experts. Even more surprisingly, the answers often don’t change much when the inputs for the computer model are put together by undergraduate students with no expertise at all.

Once I was teaching an undergraduate class at the University of Rochester while also investigating how best to get Ferdinand Marcos to resign as head of the Philippine government and create an atmosphere ripe for a free election in that country. William Casey, then Ronald Reagan’s director of intelligence, asked me to study this problem, and I was locked in a (cold) lead-lined vault at CIA headquarters to do it. I had access to classified information, but was not even allowed to read my own report when it was finished. The report was for the eyes of only the president, the vice president, the secretaries of state and defense, the national security adviser, and a few others. Meanwhile, my students worked on the same problem and were given access to the computer program I had developed to help solve such problems. They extracted the required information from magazines such as The Economist and newspapers such as The New York Times and fed their data into the computer. Ninety percent of them arrived at the same conclusions I reached in the lead-lined vault, and those conclusions about strategy proved to work rather well. This should tell us that the information needed to make good predictions is not terribly exotic; it does not require years of learning some other country’s language, history, and culture, although all of that is a big help. It should also tell us that a lot of classified information is easily reproduced from open, public sources for those who are willing to work at it.