Robert J. Moffat - Planning and Executing Credible Experiments

Physical measurements are subject to errors from many sources. Try as we may, we can never entirely correct for all the possible error sources. Sometimes we may simply decide it is too expensive to correct for a small possible error. Sometimes, we don’t know an error is present. In the best of situations, we must admit that even our corrected value is probably not exact. To deal with this situation we need a quantitative way to estimate the possible residual error and to describe it to potential users of the data. This problem was recognized by Airy (1879), who used the term “uncertainty,” which he defined as “The possible value that the residual error may have.”

Most engineering experiments involve several measurements, and the result of the experiment is derived from the measured values using a set of equations called the “data interpretation program.” The challenge is to estimate the uncertainty in the derived result as a consequence of the recognized uncertainties in the measurements.

Low uncertainty, by itself, is no assurance of accuracy in our results. The mathematical process by which we estimate the uncertainty in the result is referred to as “uncertainty analysis.”

Uncertainty analysis is a powerful tool that, properly used, can help an experimenter develop a credible experiment. For example, we can use uncertainty analysis during the planning phase in order to select the approach offering the least uncertainty. We can use it to choose the most appropriate instruments. In the “shakedown and debugging phase” of an experiment, it helps to attribute which residual errors cause differences between our result and the expected result.

Uncertainty analysis was introduced to the American technical literature in a paper by Kline and McClintock (1953). Uncertainty analysis focuses on estimating how much uncertainty there is in the derived result as a consequence of the acknowledged uncertainties in the measurements. It follows a well‐defined set of operations involving the data interpretation equations and the input data. The basic mathematics have not changed since that first description, but the techniques have been elaborated and extended considerably. See references at the end of this chapter and chapters 10, 11.

Uncertainty analysis is an essential element in experiment planning, before and during execution. Uncertainty analysis is a tool by which we can identify the significance of small changes in the output. If the observed difference is larger than the derived uncertainty, this is evidence that the process being studied is not well modeled by the equations used – in other words, something is going on that we don't know about. Chapters 10 and 11 are devoted to uncertainty analysis.

Experimentally, we never know we are “right.” An essential element to a credible experiment is acknowledging that the best we can ever say is, “We have no evidence or suspicion that we are wrong.”

Reporting the uncertainty of every reported value is essential for others to believe our science.

1 Airy, S.G.B. (1879). Theory of Errors of Observation. London, UK: Macmillan and Company.

2 Baals, D.D. and Corliss, W.R. (1981). Wind Tunnels of NASA. NASA SP‐440.

3 Baker, M. (2016). Is there a reproducibility crisis? Nature 533: 452–454.

4 Button, K.S., Ioannidis, J.P.A., Mokrysz, C. et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience.

5 Feynman, R. (1963). The Feynman Lectures on Physics, Volumes 1–3. MA: Addison‐Wesley.

6 International Astronomical Union (2006). IAU 2006 General Assembly: Result of the IAU Resolution votes. https://www.iau.org/news/pressreleases/detail/iau0603.

7 Ioannidis, J.A. (2005a). Contradicted and initially stronger effects in highly cited clinical research. JAMA 294 (2): 218–228. https://doi.org/10.1001/jama.294.2.218. PMID 16014596.

8 Ioannidis, J.P.A. (2005b). Why most published research findings are false. PLoS Medicine 2 (8): e124.

9 Ioannidis, J.P.A. (2012). Why science is not necessarily self‐correcting. Perspectives on Psychological Science 7 (6): 645–654.

10 Kline, S.J. and McClintock, F.A. (1953). Describing the uncertainties in single‐sample experiments. Mechanical Engineering 75: 3–8.

11 Muller, R. (2016). Now: The Physics of Time. NY: W.W. Norton & Company Ltd. ISBN‐13: 978‐0393285239.

12 Penrose, R. (2005). The Road to Reality: A Complete Guide to the laws of the Universe. Knopf. ISBN‐13: 978‐0679454434.

13 Penrose, R. (2016). Fashion, Faith, and Fantasy in the New Physics of the Universe. NJ: Princeton University Press. ISBN‐13: 978‐0‐691‐11979‐3.

14 Penzias, A.A. and Wilson, R.W. (1965). A Measurement of Excess Antenna Temperature at 4080 Mc/s. Astrophysical Journal 142: 419–421.

15 Pomeroy, S.R. (2012). The key to science (and life) is being wrong. Scientific American. https://blogs.scientificamerican.com/guest‐blog/the‐key‐to‐science‐and‐life‐is‐being‐wrong/.

16 Quantum Coffee (2014). Nobel Prizes in physics: Theorists vs. experimentalists. https://quantumcoffee.wordpress.com/2014/06/08/nobel‐prizes‐in‐physics‐theorists‐vs‐experimentalists

17 Simera, I., Moher, D., Hoey, J. et al. (2010). A catalogue of reporting guidelines for health research. European Journal of Clinical Investigation 40 (1): 35–53.

18 Skedung, L., Arvidsson, M., Chung, J.Y. et al. (2013). Feeling Small: Exploring the Tactile Perception Limits. Scientific Reports 3: 2617.

19 Smolin, L. (2006). The Trouble with Physics: The Rise of String Theory, the Fall of a Science, and What Comes Next. Houghton Mifflin Harcourt. ISBN‐13: 978‐0618551057.

20 Thomson, W. (1883). Wikiquote. https://en.wikiquote.org/wiki/William_Thomson

21 Tinsley, J., Molodtsov, M., Prevedel, R. et al. (2016). Direct detection of a single photon by humans. Nature Communications 7: 12172.

22 Woit, P. (2006). Not Even Wrong: The Failure of String Theory and the Continuing Challenge to Unify the Laws of Physics. Basic Books. ISBN‐13: 978‐0465092765.

1 2.1 Prepare a lab computer for dual boot: Windows and the Linux operating system. Or …

2 2.2 Prepare a dedicated lab computer for the Linux operating system. Or Exercise 2.1.

3 2.3 Following the guide in Appendix D3, download and install Gosset, public domain, open source, and free. Please consider this software tool essential.

1 1This quotation is from a Feynman lecture at Cornell in 1964. A select portion of the lecture can be viewed at https://youtu.be/OL6‐x0modwY. A lengthier quote is given in Pomeroy (2012).

2 2Rather than divide a complex number into real and imaginary parts, shall we describe it as revealed and concealed parts? Our reasoning is that both parts of a complex number have reality in our universe. For example, in dynamic systems, the revealed part corresponds to position and the concealed part corresponds to velocity. Is velocity just as real as location? Of course. We, you and us, can see the location of a car relative to its surroundings. The speed (scalar) is revealed to us via the speedometer. The change of velocity is revealed to us via our inner ear. For deeper insight, Roger Penrose has provided a most compelling motivation for complex numbers in his book Road to Reality (2005, chapters 1through 14).

3 3The superiority of the cooking profile trajectory is such that I (RH) witnessed a 50-person block of Asian tourists who each traveled to Japan primarily to buy a rice cooker to bring home to their country. There was insufficient room in the overhead compartments for all the rice cookers, so the flight departure was delayed 40 minutes as all the cookers were relocated to the checked-baggage compartment.