B. McCullough - Business Experiments with R

• The idea that small sample sizes, small effect sizes, and lots of noise can lead to false positives and even sign reversals (truly positive coefficients being estimated as negative) is discussed in terms about as nontechnical as possible in Gelman and Carlin (2014), but you'll have to know what “power” is to follow the argument, so maybe you should wait until after Chapter 2to read it.

All the web tests, in particular the results in Table 1.6, are real. Due to difficulties obtaining permissions for the original web ads, some of the web ads were mocked‐up to simulate the real ads. The photograph in Figure 1.5is from pixabay.com, and the photograph in Figure 1.7is from pexels.com.

GuessTheTest.comis a resource for digital marketers who want objective A/B test case studies and helpful information to get split‐testing ideas, insights, and best practices. There are many aspects of A/B testing on the web that are not covered in this book, and the interested reader may profitably spend some time at this website. Also, if you think you're any good at predicting the outcome of an A/B web test, to disabuse yourself of such an errant notion, try guessing at a dozen or so of the many cases presented at this website and see if you can beat 50% accuracy by a statistically significant amount.

• We barely scratched the surface of A/B testing, which, according to two recent surveys, is the most important topic in business: a survey of online marketers found “Conversion Rate Optimization” to be a top priority for the foreseeable future ( SalesForce.com, 2014); a survey of businesses that engage in conversion rate optimization used A/B testing more than any other method (Econsultancy, 2015).

• An entertaining layman's article on the rise of A/B testing can be found in Wired magazine (Christian, 2012). On A/B testing and the Obama presidential campaigns, see the interesting article in Bloomberg Businessweek by Joshua Green (2012). This is of historical interest because the Obama campaign was the first to really use analytics for fundraising and get‐out‐the‐vote activities. For those who want to learn more about the technology behind website testing and the types of tests that are possible on websites, we recommend the chapter on web testing in Waisberg and Kashuk's book titled Web Analytics 2.0 (Waisberg and Kaushik, 2009) or the succinct book by McFarland (2012) with the catchy title Experiment! .

There are also numerous other books with more in‐depth coverage of website experiments such as Siroker and Koomen (2013). Technology tools for website testing are rapidly evolving, and the evaluation of software tools is a critical first step in any website testing program. A good article on the mechanics of A/B testing on the web is Kohavi et al. (2009b). He also co‐authored an informative article, “Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained” (Kohavi et al., 2012), as well as “Seven Rules of Thumb for Web Site Experimenters” (Kohavi et al., 2014).

• An entertaining history of experimental design that gives proper due to Gosset is Ziliak (2014). To see how the methods that Gosset developed were used for commercial purposes at the Guinness Brewery, see Ziliak (2008).

• Through the 1980s, American cars didn't last long. In the 1960s, American automobile manufacturers offered warranties for 4000 miles or three months, whichever came first. By the 1990s, these warranties had been increased to 70000 miles or seven years (Lightstone et al., 1993, p. 774).

• W. E. Deming was a statistics professor at NYU in the late 1930s when he developed “statistical quality control.” He took his idea to the Detroit automakers, but they didn't want his advice. After WWII, he then took his ideas to Japan, which embraced them wholeheartedly. Deming is largely credited for the postwar Japanese economic miracle. How would the course of history have been altered, if Detroit had embraced Deming instead of rejecting him?

• As a general rule, observational results can't be trusted until they have been verified by a well‐designed experiment. Medical practices often become popular as a result of observational studies, only to be overturned years later by experiments, long after much damage has been done. Some examples are hormone replacement therapy for menopausal women, stenting for coronary disease, and a specific medicine thought to retard heart disease (fenofibrate to treat hyperlipidemia). See Huded et al. (2013) for further examples.

If these questions utterly confuse you, you probably shouldn't be reading this book.

Which of the below questions you can answer depends on the statistics course you took. You should be able to answer many of these questions. If you can answer them using the statistical software you already know but not with R, that's okay. We give R code for everything we do in this book. If you can't answer a particular question, don't worry. The methods will be explained as needed later in the book.

1 What are the following percentiles for a ‐distribution?0.900.950.975

2 For a ‐distribution, what value of gives the following proportion in the upper tail?0.010.00050.08

3 What are the following percentiles for a ‐distribution with the given degrees of freedom (df)?0.90, df=100.95, df=200.975, df=30

4 For a ‐distribution with the given degrees of freedom (df), what value of gives the following proportions in the upper tail?0.01, df=100.0005, df=200.08, df=30

5 Use the data in file SR1.csv and perform a two‐sided test of the null hypothesis that for Specifically state the non‐rejection region and the rejection region.

6 Use the data in file SR1.csv and produce a 95% confidence interval for the population mean.

7 For a two‐sided test of the null hypothesis that for , compute power for the one‐sample test of means if the true value of . Assume so that we can use the ‐table, as power calculations with the ‐table are problematic. The data are normal, and let . Compute power if . What can you infer about power? (If your first course didn't use “power,” don't worry about this one.)

8 Use the data in file SR2.csv and perform a two‐sided test of the null hypothesis that for Specifically state the non‐rejection region and the rejection region.

9 Use the data in file SR2.csv and produce a 95% confidence interval for the population proportion.

10 Use the data in file SR3.csv and compute a confidence interval for the difference between two means (hint: equal or unequal variances?).

11 Use the data in file SR3.csv and compute two confidence intervals, one for and one for . Do they overlap? What does this tell you?

12 Suppose that the data in SR3.csv are paired data, i.e. two measurements on the same experimental unit. Let be “before” and let be “after” some treatment. Perform a paired ‐test. Suppose another analyst had done a two‐sample test on these data, as in the previous two exercises. How do the analyses differ?

13 Use the data in file SR4.csv and compute a confidence interval for the difference between two proportions.

14 Use the data in file SR1.csv to make a histogram.

15 Use the data in file SR3.csv to make a scatterplot.

16 Use the data in file SR3.csv to run a simple regression, interpret the coefficients, and test whether the slope equals zero.

17 Use the data in file SR5.csv to run a multiple regression, , interpret the coefficients, and test whether a slope equals zero. (If your first course didn't cover multiple regression, don't worry about this one.) Perform model checking (i.e. check whether the assumptions are true).