B. McCullough - Business Experiments with R

• The smoother used by R in Figure 1.1is called “lowess,” which stands for LOcally WEighted Scatterplot Smoother. It's used to detect nonlinearities in a scatterplot.

• The classic reference on eliciting causal information from observational data is Rosenbaum (2010), but it is a technical book written for people who have taken advanced courses in statistics. A much more accessible book that covers both experimental and observational methods is Rosenbaum (2017), which is filled with words and has practically no equations; someone who wants to understand the topic and who hasn't taken advanced courses in statistics can do no better than this book. It also contains a very good example of how observational data can be used to inform causal questions; see the section entitled “A Simple Example: Does a Parent's Occupation Put Children at Risk?” (pp. 119‐124).

• Looking only at correlations (or lack thereof) can make it impossible to uncover causal relations. Suppose a car travels at a constant speed over hilly roads. The driver will have to accelerate on the inclines and brake on the declines to maintain a constant speed. A person who knows nothing of automobiles might observe these data and conclude that depressing the accelerator or the brake has nothing to do with what speed the car travels.

• The credit card data set is the “default of credit card clients Data Set” from https://archive.ics.uci.edu/ml/index.html. For the education variable, numbers 4, 5, and 6 were converted to “other,” similarly for the marriage variable values 0 and 4.

• It is not a good idea to run a linear regression with the variable default on the left‐hand side because default is binary (takes on only the values zero and one) and linear regression is for continuous dependent variables. There is a special method for binary dependent variables called “logistic regression,” but that's something for an advanced statistics course.

• The idea of the garden of forking paths is discussed clearly and nontechnically in Gelman and Loken (2014), which article was included in Best Math Writing of 2015 ; although nontechnical, it's an excellent read for the statistically inclined person, too.

• In general, lurking variables affect observational data and confounding variables affect designed experiments. A lurking variable connects two otherwise unconnected variables, creating the appearance of a causal relation between two other variables. Consider the firefighter example, where the number of firefighters is highly correlated with the damage caused by the fire. Adding more firefighters doesn't increase the amount of damage (the variables are really unconnected). Rather, the lurking variable “intensity of the fire” connects them. A lurking variable (say, ) creates the illusion of a causal relationship between two other variables, and . A good article on how to detect lurking variables is Joiner (1981).

• Variables are confounded when we cannot separate their respective effects on the response. A confounding variable has an effect on the response , but another variable also has an effect on , and we are unable to separate the effects of and . For example, might be store sales and is a store promotion, while is bad weather. We cannot determine the true effect of the promotion on sales because it is confounded with the weather.

Confounding can occur in a poorly designed experiment. Suppose you wish to determine the effects of fees and interest rates on credit card use. Suppose you offer a low fee and low interest rate to one group and a high fee and a high interest rate to the other group. The first group will have more credit card use and the second group less use, but you won't be able to tell whether the low fee or the low rate caused more use in the first group or whether the high fee and the high interest rate caused less use in the second group. The rate and the fee are confounded.

On the other hand, we could isolate the effect of rate by offering low fee and high rate to the first group and low fee and low rate to the second group. In more advanced designs, we sometimes will have many effects and be unable to isolate them all. In such a situation, we will deliberately confound the effects that we don't care about so much so that we can isolate the effects that we do care about. We will address this in Chapter 8.

• A layman's overview of the Salk trials is given in Meier (1989).

• The source for the polio data is http://www.post-polio.org/ir-usa.html. The source for the “under 18” US population is https://www.census.gov/data/tables/time-series/demo/popest/pre-1980-national.html. The source for the US population data is US Current Population Reports Series P25 .

• The financial services example is based on Watson‐Hemphill and Kastle (2012).

• The Progressive Insurance example comes from Chapter 22 of Holland and Cochran (2005).

• The Anheuser‐Busch example comes from Ackoff (1978).

• The number of conditions necessary to establish causality varies from discipline to discipline and even author to author. For example, the epidemiologist Hill (1965) gave nine rules. We stick with just three.

• While observational data can be useful, they are no substitute for an experiment (if an experiment can be conducted!):

But even if done perfectly, an observational study can only approach, but never reach, the credibility of randomization in assuring that there is no missing third variable that accounts for the differences observed in the experimental outcome. (Wainer, 2016, p. 48)

• The book by Schrage (2014) is entertaining and describes numerous business experiments; most of these are small, inexpensive experiments. The book by Holland and Cochran (2005), which is written for laymen, describes several larger, more complicated experiments. The article by Ganguly and Euchner (2018) describes the approach of the Goodyear Tire Company to experimentation and gives many interesting examples, including one where they conducted an experiment to determine whether tire‐pressure monitoring equipment could generate more than enough savings to pay for the monitoring equipment by reducing roadside breakdowns of tractor‐trailers. It is worthwhile to read materials such as these, for it is important for the novice experimenter to develop an idea of what has been done and what is possible.