Robert J. Moffat - Planning and Executing Credible Experiments

Richard Feynman was an experimentalist as well as a theoretician. CalTech once assigned him to teach introductory physics. His notes are immortalized in The Feynman Lectures on Physics, Volumes 1–3 (1963). He remarked about fluid physics in volume 1, chapter 3of his lectures:

There is a physical problem that is common to many fields, that is very old, and that has not been solved. It is not the problem of finding new fundamental particles, but something left over from a long time ago – over a hundred years. Nobody in physics has really been able to analyze it mathematically satisfactorily in spite of its importance to the sister sciences. It is the analysis of circulating or turbulent fluids.

In volume 2, Feynman went into more depth in chapter 41 entitled “The Flow of Wet Water.”

In a personal letter, Feynman admitted “The theory of turbulence (I have spent several years on it without success).” Turbulence was an area of classical physics beyond even Feynman's ability to solve. To this day, basic turbulence remains unsolved. As an added incentive, one of the seven Millennial Problems awaits solution of the Navier–Stokes equations which govern fluid flows.

Consider Feynman's challenge as an invitation to thermo‐fluids, experimental or theoretical. It is the most challenging area of classical physics. Turbulence becomes further complicated by heat transfer; yet more complicated by mass transfer; yet more by chemical reactions or combustion; yet more complicated by electromagnetic interactions. It is important for flight, for weather, for breath and blood, for life, for engines, for circulation within celestial stars. Flows of liquids, gases, and plasmas are found at the microscopic scale within living cells to the astronomic scales between galaxies.

One of our colleagues, Professor Adrian Bejan, overlaps with us in the same field and the same publisher. Bejan's Constructal Theory has brought a fresh theoretical approach to thermo‐fluid systems, to urban planning, and to appreciating design in living creatures as well as other fields.

At the time of the first edition, two popular TV shows featured experimental scientists. NCIS featured Abby the forensic scientist and Dr. Mallard (Ducky) the coroner; not only are they scientists, they are the most well‐rounded characters. The Big Bang Theory featured five experimentalists: Bernadette (biology), Leonard (physics), Amy (neuroscience), Raj (astrophysics), and Howard (engineer and astronaut); and one theoretician, Sheldon. The one nonscientist, Penny, was the most well rounded in reality and society.

Finally, a fun yet serious invitation. Physicists dispute a “theory of measurement” concerning mysterious quantum behavior. In certain experiments, if a measurement is made the experiment behaves one way; if no measurement is made the experimental results are opposite. The contrary results are repeatable. Include in the dispute: Is a human required as observer? Can a cat be the observer? If so, would Schrödinger's cat endanger itself? An internet search of this topic results in millions of hits, so we recommend Richard Muller's Now: The Physics of Time (2016).

Since the idea of this book is to equip you, dear Reader, to plan and execute taking measurements in your experiment, this dispute is moot.

We advise publishing and presenting your experiment plan to your client before collecting data. In the same vein, Button et al. in Nature Reviews Neuroscience (2013) made the following recommendations for researchers:

Perform an a priori power calculation. [Our note: This helps estimate sample size.]

Disclose methods and findings transparently.

Preregister your study protocol and analysis plan.

Make study materials and data available.

Work collaboratively to increase power and replicate findings.

These recommendations combined with our strategies and guidelines help to achieve a credible experiment.

If your type of experiment is a survey or poll, this text is also for you. The same strategies for sampling, statistics, and modeling apply. For a poll to promise any sort of accuracy, the sampling must be random and guided by good statistical practice. The principal challenges are designing nonleading questions and obtaining a representative sample. Otherwise a poll will mislead. Who has not witnessed and pained over a misleading poll?

Surveys of selective scientific audiences have a mixed history. One survey is particularly notable to us as experimentalists. In 2006, a survey of the International Astronomical Union (IAU) in Prague became memorable when the IAU demoted the planet Pluto. Pluto had been discovered and recognized as a planet in 1930. For decades teachers and textbooks across the world taught the nine planets in the solar system. Then the press in 2006 widely reported that the IAU declared that Pluto was no longer a planet. From our viewpoint, as IAU outsiders, it was a decree with huge impact on classrooms and the textbook industry.

As experimentalists, we are interested in the sampling, statistics, accuracy, and credibility of the Pluto survey. These all have relevance to us as we consider “The Nature of Experimental Work” (this chapter). We read that within the IAU there is dispute and controversy. How well did the vote represent the entire population?

Let's review details: The total population of IAU members is about 11,000. In 2012, the IAU reported 10,894 individual members from 93 countries worldwide. At the 2006 Prague conference, 2,400 members had registered. On the last day of the conference, just 424 astronomers remained and were polled on several issues. On some issues, exact numbers of the vote were reported (IAU 2006). However, the definitions that resulted in Pluto's demotion was reportedly decided by voice vote, approved by a majority.

How representative of the IAU population was the poll? For a sample size of 424 randomized members out of 11,000, a typical confidence interval could be the claimed 5%. The confidence interval surrounds an estimate, however a voice vote denies one. With a savvy, engaged population of size 11,000, why not take a vote of the full IAU population? Why rely on a poll to redefine international standards?

The IAU clearly has the right to set its own standards and definitions. Via the 2006 Pluto poll however, the IAU leaders decided for discriminating communication within its own community. How well did the IAU poll represent all science teachers when it demoted Pluto? By its decision, one consequence was to make existing text books obsolete, affecting the budgets of school systems worldwide.

Did the IAU leadership poorly serve its members, other scientists, and students across the world by relying on a poll instead of a full vote? From history, can we find examples where polls with small margins of error failed to predict the vote?

Another survey of note, recently conducted by the journal Nature , hearkened back to the Reproducibility Crisis announced by Ioannidis. Nature published (2016) a survey of researchers across the living and hard sciences asking “Is There a Reproducibility Crisis?” The author, Monya Baker, highlights a contradiction within the survey (Baker 2016): although researchers expected high confidence in their own fields, they admitted an inability to replicate results a majority of the time.

On a related note we may ask: what place have fashion and popularity in science? Fashion does have a notable impact on availability of funding. We defer to Panel 2.2, “Selected Invitations to Experimental Research, Insights from Theoreticians,” because funding encompasses all research, not just experiments.