Bruce Mesquita - Prediction - How to See and Shape the Future with Game Theory

What I want more than anything to show in this book, and hope that I have done so to a degree so far, is that by thinking hard about the interests involved in a given problem, we have the opportunity to take the best available steps to ensure optimal outcomes. As this next example will show, when we are unaware of the interests at play, or willfully ignore them, we can invite ruin upon ourselves.

INCENTIVIZING IGNORANCE

Arthur Andersen was driven out of business by an aggressive Justice Department looking for a big fish to fry for Enron’s bankruptcy. Later, on appeal, the Supreme Court unanimously threw out Andersen’s conviction, but it was too late to save the business. Thousands of innocent people lost their jobs, their pensions, and the pride they had in working for a successful, philanthropic, and innovative company. Andersen’s senior management apparently was entirely innocent of real wrongdoing. Unfortunately, they nevertheless helped foster their own demise by not erecting a good monitoring system to protect their business from the misbehavior of their audit clients. In fact, that was and is a problem with every major accounting firm. In Andersen’s case, I know from painful personal experience how needless their sad end was.

Around the year 2000, the head of Andersen’s risk management group asked me if I could develop a game-theory model that would help them anticipate the risk that some of their audit clients might commit fraud (this is where my work related to the Sarbanes-Oxley discussion from a few chapters back began). As I have related, three colleagues and I constructed a model to predict the chances that a company would falsely report its performance to shareholders and the SEC. Our game-theory approach, coupled with publicly available data, makes it possible to predict the likelihood of fraud two years in advance of its commission. We worked out a way to identify a detailed forensic accounting that helps assess the likely cause of fraud—if any—as a function of any publicly traded company’s governance structure.

We grouped companies according to the degree to which our model projected that they were at risk of committing fraud. Of all the firms we examined, 98 percent were predicted to have a near-zero risk of committing fraud. Barely 1 percent of those firms were subsequently alleged to have reported their performance fraudulently. At the other end of our scale, about 1.5 percent of companies were placed in the highest risk category based on the corporate organizational and compensation factors assessed by the model. A whopping 85 percent of that small group of companies were accused by the SEC of committing fraud within the time window investigated by the model. This is a very effective system that produces few false positives—alleging that a company would commit fraud when it apparently did not—and very few false negatives—suggesting that a company would not commit fraud when it subsequently did.

Enron was one of the 1.5 percent of companies that we highlighted as being in the highest risk category. You can see this in the table on page 119, which shows our predictions for a select group of companies that eventually were accused of very big frauds. The table shows our assessment of the risk of fraud for each company each year. The estimates of interest are for 1997–99. These assessments are based on what is called in statistics an out-of-sample test. Let me explain what that means and how it is constructed.

Suppose you want to know how likely it is that a company is in either of two categories: honest or fraudulent. Using game-theory reasoning, you might identify several factors that nudge executives to resort to fraud when their company is in trouble. A few chapters back we talked about some of those factors, such as the size of the group of people whose support executives need to keep their jobs, and we talked about factors that provide early-warning signs of fraud, such as dividend and management compensation packages that are below expectations given the reported performance of the firm and its governance structure.

We know that some conditions, including the amount paid out in dividends, indicate whether fraud is more or less likely; but how important is the magnitude of dividend payments in influencing the risk of fraud compared to, for instance, the percentage of the company owned by large institutional investors? That too is an important indicator of the incentive to hide or reveal poor corporate performance. There are statistical procedures that evaluate the information on many variables (the factors identified in the fraud game devised by my colleagues and me, for example) to work out how well those factors predict the odds that a company is honest or fraudulent (or whatever else it is that is being studied).

There is a family of statistical methods known as maximum likelihood estimation for doing this. We won’t worry here about exactly how these methods work. (For the aficionados, we used logit analysis.) The important thing is that these methods produce unbiased estimates of the relative weight or importance of each of the factors, each of the variables, thought to influence the outcome. By multiplying each factor’s value (the number of directors, for example, or the percentage of the firm owned by institutional investors) by its weight, we can get a composite estimate of the probability that the firm will be honest or will commit fraud two years in the future. If the theory is just plain wrong, then these statistical methods will show that the factors in the equation do not significantly influence whether a firm is honest or fraudulent in the way the theory predicts.

The weights we estimated were derived from data on hundreds of companies for the years 1989 through 1996. Since the thing we were interested in predicting—corporate honesty or fraud—was unknown for 1997 in 1995, or for 1998 in 1996, or for 1999 in 1997, and so forth, the statistically estimated weights were limited to just those years for which we knew the outcomes as well as the inputs from two years earlier. Thus, the last year for which we used the statistical method to fit data to a known outcome was 1996. We then applied the weights created by the in-sample test to estimate the likelihood of fraud for the years of data that were not included in our statistical calculation. Those years are the out-of-sample cases. The out-of-sample predictions, then, cover the years 1997 forward in the table. Of course, since this analysis was actually being done in 2000 and 2001, we were “predicting” the past.

Now, you may well think this is an odd view of prediction. It is unlike anything I have discussed so far. What, you might wonder, does it mean to predict the past? That must be very easy when you already know what happened. But remember, in the out-of-sample test, nothing that happened after 1996 was utilized to create variable weights or to pick the variables that were important. Since the predictions about events after 1996 took no advantage of any information after that year, they are true predictions even though they were created in 2000 and 2001. This sort of out-of-sample test is useful to assess whether our model worked effectively at distinguishing between companies facing a high and a low risk of fraud. That is not to say that it is useful from a practical standpoint, even though it is useful from the perspective of validating the model and in providing confidence about how it could be expected to perform in the future. Let me explain what I mean:

Predicting the past can be helpful in terms of advancing science, even though it is not of much practical use when it comes to avoiding the audit of firms that have already committed fraud. But if it accurately predicted the pattern of fraud in the past, it is likely to do the same in the future.