B. McCullough - Business Experiments with R

Library of Congress Cataloging‐in‐Publication Data

Names: McCullough, Bruce D., author.

Title: Business experiments with R / Bruce. D. McCullough, Drexel

University.

Description: Second edition. | Hoboken : Wiley, 2021. | Includes index.

Identifiers: LCCN 2020029253 (print) | LCCN 2020029254 (ebook) | ISBN

9781119689706 (hardback) | ISBN 9781119689904 (adobe pdf) | ISBN

9781119689881 (epub)

Subjects: LCSH: Commercial statistics.

Classification: LCC HF1017 .M366 2021 (print) | LCC HF1017 (ebook) | DDC

658.4/034–dc23

LC record available at https://lccn.loc.gov/2020029253

LC ebook record available at https://lccn.loc.gov/2020029254

Cover Design: Wiley

Cover Image: © MR.Cole_Photographer/Getty Images

To my lovely wife and wonderful children

Rigorous experiments are rapidly becoming a critical tool for businesses. While experiments have been a popular tool in engineering for decades, driven largely by the quality movement of the 1980s, experiments are rapidly coming out of the engineering lab to be deployed more broadly across business operations. Companies today are using experiments to solve a broad range of problems from determining whether improved training results in better employee productivity and retention to gauging whether store renovations result in higher profitability, to determining which subject lines in a promotional email attract customer attention, to assessing which behavioral nudges are most likely to get utility users to shift energy consumption to nonpeak times, and to deciding which color to use for the checkout buttons on a website. Articles in the business press such as those by Davenport (2009), Anderson and Simester (2011), and Thomke and Manzi (2014) have made the case that experiments are a key component of “data‐driven” business and companies are taking note and incorporate experiments into their day‐to‐day operations. With this rapid adoption of experimentation, there is a need for a comprehensive resource for business leaders and analysts who want to use experiments to understand how business systems work and solve tactical business problems.

Yet, material on business experiments remains siloed within specific business functions. For example, there are dozens of books and articles on website testing that focus intently on experimentation with websites and never mention that the same basic ideas – randomization, measurement, and comparison – can be applied in other business domains. Our goal for this book is to step back from this highly tactical approach and give business students a broad, theoretically grounded understanding of the role of experiments in business, providing them with a practical foundation for designing and analyzing experiments that will be useful to them throughout their careers.

Of course, experiments have long been a foundational tool in engineering, physical sciences, psychology, medicine, and statistics, and excellent books on how to design and analyze experiments are available within each of these fields, yet none of these books provides the right coverage for business students. The literature on experiments in the physical sciences and engineering provides great depth on multivariable test design, yet often focuses on situations where the experimenter is studying a physical process in a lab setting, where everything can be measured and controlled and error can be reduced nearly to zero – conditions that seldom hold in business. The classic textbook on experimental design for engineers by Montgomery (2017) naturally assumes a level mathematical skill that is not common among business students. Psychologists like Campbell and Stanley (1963) focus more on experiments with human subjects, raising many of the same issues that come up in marketing and human resources experiments, yet they focus intently on using experiments to develop theory and so cover issues like construct validity that are less relevant in the types of tactical experiments that business uses to drive day‐to‐day decisions. Useful discussions of the practicalities of running experiments in the field have appeared recently in the economics literature championed by Harrison and List (2004) and Gerber and Green (2012), yet the experiments reported tend to focus on testing economic theory. In many ways, the types of tactical experiments that are valuable in business are much more similar to clinical trials and other biomedical experiments, where there is a tradition of tactical experiments designed to determine, “Should we give this treatment to patients?” and a well‐established culture of randomized controlled trials as the gold standard of evidence. While much of this work from other disciplines is relevant to business experiments, it doesn't provide students with concrete examples of how to apply these methods to answer important business questions.

This book has drawn from all of these literatures selecting the key ideas that business students need to know about experiments. This book approaches experimentation as a skill that one becomes better at over time, like playing a sport or an instrument, which requires equal measures of creativity and technical skill. To help students practice, the book is structured around a series of examples, focusing first on a specific business question, then turning to the design or analysis of an experiment that then will answer that question, and then building up the appropriate statistical methods. This way, the technical material is presented in context. This approach provides an ideal “second course in statistics,” where students can review material they have seen before (specifically two‐sample tests, power, ANOVA, confidence intervals, and regression) in the context of business problems. My hope is that students who work through this book will move beyond the mechanics and develop their abilities to frame ill‐defined problems, determine what data and analysis would provide information about that problem, and examine the evidence for or against a particular business decision.

For clarity in the use of pronouns, I have adopted the convention that the analyst is always female. Other persons are male.

This book can be used both with general business students, such as MBA or executive education students, and with the growing number of master's programs in business analytics and marketing analytics.

The first four or five chapters (depending on instructor preference) provides students with the tools they need to design and analyze two‐treatment experiments (i.e. A/B tests) to answer business questions, focusing on the strategic and technical issues involved in designing experiments that will truly affect organizations. The book begins with a discussion of the importance of causal analysis in business and the risks of observational data analysis in Chapter 1. By design, the next chapter begins with the analysis of A/B tests, with a review of the statistical methods required for analyzing experiments, focusing on visualization and hypothesis testing. Once students understand how experiments are analyzed, in Chapter 3, they can move on to understanding issues involved in designing experiments including selecting treatments, measuring responses, planning sample sizes, and the importance of randomization for causal interpretation. Chapter 4returns to the analysis of A/B tests, covering several advanced analysis techniques including analyzing tests with matched pairs, analyzing tests with more than two treatment groups, analyzing subgroups within an A/B test, and determining the minimal detectable effect for an experiment that has been completed. Chapter 5, which covers issues involved in the design of A/B tests when the number of units available for tests is small, as is the case when doing geo‐testing or testing across store locations. This coverage makes Part I ideal for a half‐semester graduate course for second‐year MBA students or strong undergraduate students. These chapters can be covered over five weeks, skipping some of the more technical material and allowing time for the students to apply the material to a project where they plan, execute, and analyze an A/B test.