Douglas W. Hubbard - The Failure of Risk Management

EXHIBIT 2.1 Current Top Risks According to Three Surveys

Respondents would mostly say their methods are “formal:” The 2017 Aon study found that 60 percent state they have adopted formal or partially formal approaches to risk management. The share that say they have a formalized risk management approach goes up with the size of the firm—96 percent of firms with revenue over $10 billion say they use a formalized approach. About 70 percent overall would claim to have a formal or partially formal approach.

Formal mostly means “qualitative procedure” not quantitative: The HDR/KPMG survey found that what these $10 billion firms mean by formal is mostly (74 percent) a qualitative ranking or scoring method, perhaps using a form of the qualitative risk matrix. This is about the same for companies under that revenue threshold (78 percent). Only 16 percent of firms with revenue over $10 billion (and 20 percent of firms of all sizes) say they use quantitative methods—that is, they use explicit probabilities derived from mathematical and empirical methods using tools such as simulations and tools familiar to actuaries, statisticians, or quantitative risk analysts. Of those who use quantitative methods, the most common is Monte Carlo simulations (85 percent) followed by statistical analysis of historical data (77 percent). Less common are methods such as Bayesian statistics (56 percent) or utility theory (17 percent).

There are obstacles to the adoption of quantitative methods, but adoption is feasible: In the 2007 Protiviti survey, 57 percent said they quantify risks “to the fullest extent possible,” up from 41 percent in 2006. Because, as we noted, only 20 percent of all firms use some form of actual probabilistic methods, it would seem that most respondents in the Protiviti survey would not consider these methods possible. In fact, our survey found that 42 percent said an obstacle to the adoption of quantitative methods was “skepticism about the practicality and effectiveness.” Yet our survey showed that those who use quantitative methods such as simulations and statistical methods come from a variety of industries and company sizes. Even though quantitative methods are common in some industries (finance, insurance, etc.), the users outside of those industries are arguably as diverse as the users of qualitative methods. Apparently, there will be active users of these methods in the same industries and contexts where there are also skeptics.

These surveys agree with my personal experience on some key points. I see that most organizations who say they follow a formal method are merely saying they follow a defined procedure . Whether that defined procedure is based on mathematically and scientifically sound principles—what has been measured to work—is another question altogether. (More on that later.) Exhibit 2.2provides a summary of what risk assessment methods are used, according to the HDR/KPMG survey.

Each of the categories in exhibit 2.2contains many specific variations. So, let's dive into each of them in more detail.

EXHIBIT 2.2 Summary of Risk Assessment Methods Used According to the HDR/KPMG Survey

The most basic of these is part of the “everything else” category in exhibit 2.2— expert intuition . This is a sort of baseline of risk management methods. This is pure gut feel unencumbered by structured rating or evaluation systems of any kind. There are no points, probabilities, scales, or even standardized categories. There are shortcomings to this but there is also lot of value. Experts do know something, especially if we can adjust for various biases and common errors. In order for other methods to be of any value at all, they must show a measurable improvement on gut feel. (In fact, we will show later that unaided expert intuition isn't the worst of them.)

Other approaches that we lumped into the “everything else” category are various forms of audits and checklists. They don't do any structured prioritization of risks based on real measurements. They just make sure you don't forget something important and systematically search for problems. You definitely want your pilot and surgeon to use checklists and to guard against fraud or mistakes; you want your firm's books to be audited. I mention them here because it could be argued that checklists sometimes perform a pure assessment role in risk management. Most organizations will use audits and checklists of some sort even if they don't fall under the sort of issues risk managers may concern themselves with.

The most common risk assessment method is some form of a risk matrix. A total of 41 percent of respondents in the HDR/KPMG survey say they use a risk matrix—14 percent use a risk matrix based on one of the major standards (e.g., NIST, ISO, COSO, etc.) and 27 percent use an internally developed risk matrix. Internally developed risk matrices are most common in firms with revenue over $10 billion, where 39 percent say that is the method they use.

Risk matrices are among the simplest of the risk assessment methods and this is one reason they are popular. Sometimes referred to as heat map or risk map, they also provide the type of visual display often considered necessary for communication to upper management. See exhibit 2.3for an example of a risk map for both verbal categories and numerical scores.

As the exhibit shows, a risk matrix has two dimensions, usually labeled as likelihood on one axis and an impact on the other. Typically, likelihood and impact are then evaluated on a scale with verbal labels. For example, different levels of likelihood might be called likely, unlikely, extremely unlikely, and so on. Impact might be moderate or critical. Sometimes, the scales are numbered, most commonly on a scale of 1 to 5, where 1 is the lowest value for likelihood or impact and 5 is the highest. Sometimes these scores are multiplied together to get a “risk score” between 1 and 25. The risk matrix is often further divided into zones where total risk, as a function of likelihood and impact, is classified as high-medium-low or red-yellow-green.

EXHIBIT 2.3 Does This Work? One Version of a Risk Map Using Either Numerical or Verbal Scales

There are many variations of risk matrices in many fields. They may differ in the verbal labels used, the point scale, whether the point scales are themselves defined quantitatively, and so on. Chapter 8will have a lot more on this.