Robert J. Moffat - Planning and Executing Credible Experiments

1 Cover

2 Title Page

3 Copyright Page

4 Dedication Page

5 About the Authors

6 Preface Audience Accompanying Material Recommended Companion Texts

7 Acknowledgments

8 About the Companion Website

9 1 Choosing Credibility 1.1 The Responsibility of an Experimentalist 1.2 Losses of Credibility 1.3 Recovering Credibility 1.4 Starting with a Sharp Axe 1.5 A Systems View of Experimental Work 1.6 In Defense of Being a Generalist References Homework

10 2 The Nature of Experimental Work 2.1 Tested Guide of Strategy and Tactics 2.2 What Can Be Measured and What Cannot? 2.3 Beware Measuring Without Understanding: Warnings from History 2.4 How Does Experimental Work Differ from Theory and Analysis? 2.5 Uncertainty 2.6 Uncertainty Analysis References Homework

11 3 An Overview of Experiment Planning 3.1 Steps in an Experimental Plan 3.2 Iteration and Refinement 3.3 Risk Assessment/Risk Abatement 3.4 Questions to Guide Planning of an Experiment Homework

12 4 Identifying the Motivating Question 4.1 The Prime Need 4.2 An Anchor and a Sieve 4.3 Identifying the Motivating Question Clarifies Thinking 4.4 Three Levels of Questions 4.5 Strong Inference 4.6 Agree on the Form of an Acceptable Answer 4.7 Specify the Allowable Uncertainty 4.8 Final Closure Reference Homework

13 5 Choosing the Approach5.1 Laying Groundwork 5.2 Experiment Classifications 5.3 Real or Simplified Conditions? 5.4 Single‐Sample or Multiple‐Sample? 5.5 Statistical or Parametric Experiment Design? 5.6 Supportive or Refutative? 5.7 The Bottom Line References Homework

14 6 Mapping for Safety, Operation, and Results6.1 Construct Multiple Maps to Illustrate and Guide Experiment Plan 6.2 Mapping Prior Work and Proposed Work 6.3 Mapping the Operable Domain of an Apparatus 6.4 Mapping in Operator's Coordinates 6.5 Mapping the Response Surface

15 7 Refreshing Statistics 7.1 Reviving Key Terms to Quantify Uncertainty 7.2 The Data Distribution Most Commonly EncounteredThe Normal Distribution for Samples of Infinite Size 7.3 Account for Small Samples: The t‐Distribution 7.4 Construct Simple Models by Computer to Explain the Data 7.5 Gain Confidence and Skill at Statistical Modeling Via the R Language 7.6 Report Uncertainty 7.7 Decrease Uncertainty (Improve Credibility) by Isolating Distinct Groups 7.8 Original Data, Summary, and R References Homework

16 8 Exploring Statistical Design of Experiments8.1 Always Seeking Wiser Strategies 8.2 Evolving from Novice Experiment Design 8.3 Two‐Level and Three‐Level Factorial Experiment Plans 8.4 A Three‐Level, Three‐Factor Design 8.5 The Plackett–Burman 12‐Run Screening Design 8.6 Details About Analysis of Statistically Designed Experiments 8.7 Retrospect of Statistical Design Examples 8.8 Philosophy of Statistical Design 8.9 Statistical Design for Conditions That Challenge Factorial Designs 8.10 A Highly Recommended Tool for Statistical Design of Experiments 8.11 More Tools for Statistical Design of Experiments 8.12 Conclusion Further Reading Homework

17 9 Selecting the Data Points 9.1 The Three Categories of Data 9.2 Populating the Operating Volume 9.3 Example from Velocimetry 9.4 Organize the Data 9.5 Strategies to Select Next Data Points 9.6 Demonstrate Gosset for Selecting Data 9.7 Use Gosset to Analyze Results 9.8 Other Options and Features of Gosset 9.9 Using Gosset to Find Local Extrema in a Function of Several Variables 9.10 Summary Further Reading Homework

18 10 Analyzing Measurement Uncertainty10.1 Clarifying Uncertainty Analysis 10.2 Definitions 10.3 The Sources and Types of Errors 10.4 The Basic Mathematics 10.5 Automating the Uncertainty Analysis 10.6 Single‐Sample Uncertainty Analysis References Further Reading Homework

19 11 Using Uncertainty Analysis in Planning and Execution 11.1 Using Uncertainty Analysis in Planning 11.2 Perform Uncertainty Analysis on the DREs 11.3 Using Uncertainty Analysis in Selecting Instruments 11.4 Using Uncertainty Analysis in Debugging an Experiment 11.5 Reporting the Uncertainties in an Experiment 11.6 Multiple‐Sample Uncertainty Analysis 11.7 Coordinate with Uncertainty Analysis Standards 11.8 Describing the Overall Uncertainty in a Single Measurement 11.9 Additional Statistical Tools and Elements References Homework

20 12 Debugging an Experiment, Shakedown, and Validation12.1 Introduction 12.2 Classes of Error 12.3 Using Time‐Series Analysis in Debugging 12.4 Examples 12.5 Process Unsteadiness 12.6 The Effect of Time‐Constant Mismatching 12.7 Using Uncertainty Analysis in Debugging an Experiment 12.8 Debugging the Experiment via the Data Interpretation Program 12.9 Situational Uncertainty

21 13 Trimming Uncertainty13.1 Focusing on the Goal 13.2 A Mlotivating Question for Industrial Production 13.3 Plackett–Burman 12‐Run Results and Motivating Question #3 13.4 PB 12‐Run Results and Motivating Question #1 13.5 Uncertainty Analysis of Dual Predictive Model and Motivating Question #2 13.6 The PB 12‐Run Results and Individual Machine Models 13.7 Final Answers to All Motivating Questions for the PB Example Experiment 13.8 Conclusions

22 14 Documenting the Experiment 14.1 The Logbook 14.2 Report Writing 14.3 International Organization for Standardization, ISO 9000 and other Standards 14.4 Never Forget. Always Remember

23 Appendix A: Distributing Variation and Pooled VarianceA.1 Inescapable Distributions A.2 Other Common Distributions A.3 Pooled Variance (Advanced Topic)

24 Appendix B: Illustrative Tables for Statistical DesignB.1 Useful Tables for Statistical Design of Experiments B.2 The Plackett–Burman (PB) Screening Designs

25 Appendix C: Hand Analysis of Two‐Level Factorial Designs C.1 The General Two‐Level Factorial Design C.2 Estimating the Significance of the Apparent Factor Effects C.3 Hand Analysis of a Plackett–Burman (PB) 12‐Run Design C.4 Illustrative Practice Example for the PB 12‐Run Pattern C.5 Answer Key: Compare Your Hand Calculations C.6 Equations for Hand Calculations

26 Appendix D: Free Recommended Software: Obtain Recommended Free, Open‐Source Software for Your ComputerD.1 Instructions to Obtain the R Language for Statistics D.2 Instructions to Obtain LibreOffice D.3 Instructions to Obtain Gosset D.4 Possible Use of RStudio

27 Index

28 End User License Agreement

1 Chapter 3 Table 3.1 Overview of a research experiment plan. Table 3.2 Review the program plan. Do risk assessment and plan risk abatement. I... Table 3.3 Assess the credibility of the program. Do risk assessment and plan ...

2 Chapter 6Table 6.1 Stanton number as a function of blowing fraction, F.Table 6.2 Data format for computer and statistical analysis: text example.

3 Chapter 7Table 7.1 Economic experiment example for shoe store.Table 7.2 t‐Distribution.

4 Chapter 8Table 8.1 Example 1, Part 1: codebook of test conditions for a three‐level, t...Table 8.2 Example l, Part 2: coding and assignment of trial identification nu...Table 8.3 Example 1, Part 3: the randomized trial schedule and test results.Table 8.4 Example 1, Part 4: Analysis of Variance of the modeling terms.Table 8.5 Example 2, Part 1: Codebook of factors and codings for a six‐factor...Table 8.6 Example 2, Part 2: 12‐run PB design for a six‐ parameter problem.Table 8.7 Example 2, Part 3: Analysis of Variance as parsed for each model te...

5 Chapter 9Table 9.1 Codebook for life jacket safety spreadsheet (total versus partial r...Table 9.2 List of factors for industrial test plan example, plus material pro...Table 9.3 PB‐guided initial 12 runs of industrial example.Table 9.4 Analysis of Variance of factors for the full first‐order linear mod...

6 Chapter 11Table 11.1 Sensitivity coefficients.Table 11.2 Probability points of the double‐sided student's t‐distribution.Table 11.3 Estimating sigma from S N.