Douglas W. Hubbard - The Failure of Risk Management

1 Cover

2 About the Author About the Author Mr. Hubbard's career in quantitatively based management consulting began in 1988 with Coopers & Lybrand. He founded Hubbard Decision Research in 1999 and he developed the applied information economics (AIE) method to solve complex, risky decisions. He has used AIE in many fields, including cybersecurity, aerospace, biotech, environmental policy, commercial real estate, tech startups, entertainment, and military logistics, to name a few. His AIE methodology has received critical praise from respected research firms such as Gartner, Forrester, and others. He is the author of the following books (all published with Wiley between 2007 and 2016): How to Measure Anything: Finding the Value of Intangibles in Business (one of the all-time, best-selling books in business math) The Failure of Risk Management: Why It's Broken and How to Fix It Pulse: The New Science of Harnessing Internet Buzz to Track Threats and Opportunities How to Measure Anything in Cybersecurity Risk (coauthored with Richard Seiersen) His books have sold over 140,000 copies in eight languages and are used as textbooks in dozens of university courses including at the graduate level. Two of his books are required reading for the Society of Actuaries exam prep, and he is the only author with more than one on the list. In addition to his books, Mr. Hubbard has published articles in Nature, The American Statistician, IBM Journal of R&D, CIO Magazine , and more.

3 Preface

4 Acknowledgments

5 PART ONE: An Introduction to the Crisis CHAPTER 1: Healthy Skepticism for Risk Management A “COMMON MODE FAILURE” KEY DEFINITIONS: RISK MANAGEMENT AND SOME RELATED TERMS WHAT FAILURE MEANS SCOPE AND OBJECTIVES OF THIS BOOK NOTES CHAPTER 2: A Summary of the Current State of Risk Management A SHORT AND ENTIRELY-TOO-SUPERFICIAL HISTORY OF RISK CURRENT STATE OF RISK MANAGEMENT IN THE ORGANIZATION CURRENT RISKS AND HOW THEY ARE ASSESSED NOTES CHAPTER 3: How Do We Know What Works? ANECDOTE: THE RISK OF OUTSOURCING DRUG MANUFACTURING WHY IT'S HARD TO KNOW WHAT WORKS AN ASSESSMENT OF SELF-ASSESSMENTS POTENTIAL OBJECTIVE EVALUATIONS OF RISK MANAGEMENT WHAT WE MAY FIND NOTES CHAPTER 4: Getting Started: A Simple Straw Man Quantitative Model A SIMPLE ONE-FOR-ONE SUBSTITUTION THE EXPERT AS THE INSTRUMENT A QUICK OVERVIEW OF “UNCERTAINTY MATH” ESTABLISHING RISK TOLERANCE SUPPORTING THE DECISION: A RETURN ON MITIGATION MAKING THE STRAW MAN BETTER NOTE

6 PART TWO: Why It's Broken CHAPTER 5: The “Four Horsemen” of Risk Management: Some (Mostly) Sincere Attempts to Prevent an Apocalypse ACTUARIES WAR QUANTS: HOW WORLD WAR II CHANGED RISK ANALYSIS FOREVER ECONOMISTS MANAGEMENT CONSULTING: HOW A POWER TIE AND A GOOD PITCH CHANGED RISK MANAGEMENT COMPARING THE HORSEMEN MAJOR RISK MANAGEMENT PROBLEMS TO BE ADDRESSED NOTES CHAPTER 6: An Ivory Tower of Babel: Fixing the Confusion about Risk THE FRANK KNIGHT DEFINITION KNIGHT'S INFLUENCE IN FINANCE AND PROJECT MANAGEMENT A CONSTRUCTION ENGINEERING DEFINITION RISK AS EXPECTED LOSS DEFINING RISK TOLERANCE DEFINING PROBABILITY ENRICHING THE LEXICON NOTES CHAPTER 7: The Limits of Expert Knowledge: Why We Don't Know What We Think We Know about Uncertainty THE RIGHT STUFF: HOW A GROUP OF PSYCHOLOGISTS MIGHT SAVE RISK ANALYSIS MENTAL MATH: WHY WE SHOULDN'T TRUST THE NUMBERS IN OUR HEADS “CATASTROPHIC” OVERCONFIDENCE THE MIND OF “ACES”: POSSIBLE CAUSES AND CONSEQUENCES OF OVERCONFIDENCE INCONSISTENCIES AND ARTIFACTS: WHAT SHOULDN'T MATTER DOES ANSWERS TO CALIBRATION TESTS NOTES CHAPTER 8: Worse Than Useless: The Most Popular Risk Assessment Method and Why It Doesn't Work A FEW EXAMPLES OF SCORES AND MATRICES DOES THAT COME IN “MEDIUM”?: WHY AMBIGUITY DOES NOT OFFSET UNCERTAINTY UNINTENDED EFFECTS OF SCALES: WHAT YOU DON'T KNOW CAN HURT YOU DIFFERENT BUT SIMILAR-SOUNDING METHODS AND SIMILAR BUT DIFFERENT-SOUNDING METHODS NOTES CHAPTER 9: Bears, Swans and Other Obstacles to Improved Risk Management ALGORITHM AVERSION AND A KEY FALLACY ALGORITHMS VERSUS EXPERTS: GENERALIZING THE FINDINGS A NOTE ABOUT BLACK SWANS MAJOR MATHEMATICAL MISCONCEPTIONS WE'RE SPECIAL: THE BELIEF THAT RISK ANALYSIS MIGHT WORK, BUT NOT HERE NOTES CHAPTER 10: Where Even the Quants Go Wrong: Common and Fundamental Errors in Quantitative Models A SURVEY OF ANALYSTS USING MONTE CARLOS THE RISK PARADOX FINANCIAL MODELS AND THE SHAPE OF DISASTER: WHY NORMAL ISN'T SO NORMAL FOLLOWING YOUR INNER COW: THE PROBLEM WITH CORRELATIONS THE MEASUREMENT INVERSION IS MONTE CARLO TOO COMPLICATED? NOTES

7 PART THREE: How to Fix It CHAPTER 11: Starting with What Works SPEAK THE LANGUAGE GETTING YOUR PROBABILITIES CALIBRATED USING DATA FOR INITIAL BENCHMARKS CHECKING THE SUBSTITUTION SIMPLE RISK MANAGEMENT NOTES CHAPTER 12: Improving the Model EMPIRICAL INPUTS ADDING DETAIL TO THE MODEL ADVANCED METHODS FOR IMPROVING EXPERT'S SUBJECTIVE ESTIMATES OTHER MONTE CARLO TOOLS SELF-EXAMINATIONS FOR MODELERS NOTES CHAPTER 13: The Risk Community: Intra- and Extra-organizational Issues of Risk Management GETTING ORGANIZED MANAGING THE MODEL INCENTIVES FOR A CALIBRATED CULTURE EXTRAORGANIZATIONAL ISSUES: SOLUTIONS BEYOND YOUR OFFICE BUILDING PRACTICAL OBSERVATIONS FROM TRUSTMARK FINAL THOUGHTS ON QUANTITATIVE MODELS AND BETTER DECISIONS NOTES

8 APPENDIX: Additional Calibration Tests and Answers

9 Index

10 End User License Agreement

1 Chapter 2 EXHIBIT 2.1 Current Top Risks According to Three Surveys EXHIBIT 2.2 Summary of Risk Assessment Methods Used According to the HDR/KPM...

2 Chapter 4 EXHIBIT 4.1 Simple Substitution of Quantitative versus the Risk Matrix

3 Chapter 5EXHIBIT 5.2 Summary of the Four Horsemen

4 Chapter 6EXHIBIT 6.1 Example of the Risk of a Project Failure Expressed as a Vector Q...EXHIBIT 6.3 Example of Three Bets Considered Equivalent by a Decision-Maker

5 Chapter 7EXHIBIT 7.1 Sample Calibration Test with True/False Trivia Questions EXHIBIT 7.3 Sample Calibration Test with 90 Percent Confidence Interval* Que...

6 Chapter 8EXHIBIT 8.1 Likelihood Scale Proposed by the National Institute for Standards ...EXHIBIT 8.2 Impact Scale Proposed by the National Institute for Standards ...EXHIBIT 8.3 Variances in Understanding of Common Terms Used in the IPCC Repo...EXHIBIT 8.6 Examples of Impact Scale Partitions

7 Chapter 11EXHIBIT 11.2 LEC vs. Risk Tolerance

8 Chapter 12EXHIBIT 12.4 Some Available Monte Carlo Tools

9 Chapter 13EXHIBIT 13.2 Brier's Core Example for Forecasters

1 Chapter 2 EXHIBIT 2.3 Does This Work? One Version of a Risk Map Using Either Numerical...

2 Chapter 4 EXHIBIT 4.2 Example of a Loss Exceedance Curve

3 Chapter 5EXHIBIT 5.1 Risk and Probability in Economic Literature

4 Chapter 6EXHIBIT 6.2 The Exponential Utility Function Where S = $5 Million

5 Chapter 7EXHIBIT 7.2 Comparison of Uncalibrated and Calibrated IndividualsEXHIBIT 7.4 Score Distribution of Actual Ten-Question 90 Percent Confidence ...EXHIBIT 7.5 Agreement between First and Second Estimates of IT Project Failu...

6 Chapter 8EXHIBIT 8.4 Distribution of Scores from Five Different Five-Point ScalesEXHIBIT 8.5 Comparison of Scales versus Historical Data

7 Chapter 10EXHIBIT 10.1 The Normal DistributionEXHIBIT 10.2 Power-Law Distributions of Hurricane and Earthquake Frequency a...EXHIBIT 10.3 Frequency and Magnitude of Daily Price Drops in the S&P 500 and...EXHIBIT 10.4 Examples of Correlated DataEXHIBIT 10.5 Same Correlation Coefficient, Very Different Patterns

8 Chapter 11EXHIBIT 11.1 Mapping the Nonquantitative Matrix to a Simple Quantitative Rep...EXHIBIT 11.3 A (Simplified) Risk Management Cycle

9 Chapter 12EXHIBIT 12.1 Application of Robust Bayesian Method to Launch Failure RatesEXHIBIT 12.2 Conditional Robust Bayesian Method: Chance of Shuttle Disaster ...EXHIBIT 12.3 Examples of Various Distribution Types