Robert J. Moffat - Planning and Executing Credible Experiments

Be aware. Beware small studies (i) and small effects (ii). Beware explaining indiscriminately (iii). Beware lack of standards (iv). Beware political favors and conflicts of interest (v). Beware fashionable science (vi). The corollaries provide fair warning for all research, and do affect the Nature of Experimental Work.

Feynman's words suggest that experiments help science to be self‐correcting. However, Ioannidis (2012) gave the warning “Why Science Is Not Necessarily Self‐Correcting” in a more recent paper concerning psychological science.

Dr. J.P.A. Ioannidis has done more for us than exposing flaws in medical science. He invites better experiments in the biological and medical sciences. He sharpens experimental discernment.

Dr. Ioannidis is not alone. For example, Simera et al. (2010) report in the European Journal of Clinical Investigation how researchers collaborated and top publishers have agreed to guidelines regarding health research studies. There are many similar groups now.

Furthermore, there is incentive to resist bad and fraudulent research. RetractionWatch.comkeeps track of research that has been retracted by author, publisher, or sponsor.

Overall, the concerted effort to assure the credibility of experiments keeps us in good company.

The desire to better understand how our world works invites us to credible experiments. Beyond measuring, we predict what we expect to measure. Our interest may be our company's factory floor, economics and marketing, agriculture and environment, medicine, information, engineering and technology, climate, or the sciences biology, chemistry, and physics.

Richard Feynman in his lecture “There's Plenty of Room at the Bottom” (1959) invited scientists into a new field which we now call “nanoscience.” The launch of nanoscience has led to innovative products that affect our daily lives. Nanoscience, as a research field, reaches from measuring individual atoms within molecules, to manipulating atoms, to fluid nano‐arrays for medical and pharmaceutical tests and beyond.

My (RH) first experimental mentor in aerodynamics was W.S. Saric. He told me the advantage our area had over other physics fields: in thermo‐fluid physics, researchers shared their techniques. His reasoning: since Ludwig Prandtl, an early pioneer in fluids, had not won a Nobel Prize, no one in our area of classical physics would expect to win. Classic thermo‐fluid physics was crucial to so many areas of science. The camaraderie of joint effort was inviting.

Essentially every Nobel Prize in Chemistry recognizes experimental work. Likewise, essentially every Nobel Prize in Physiology or Medicine recognizes experimental tests. How about for Physics Nobel Prizes? An accounting (Quantum Coffee 2014) of the Nobel Prizes in Physics up to 2014 divided as such: theory, 30.75 prizes (28.7%); experiment, 76.25 prizes (71.3%). Nobel Prizes in Physics for technical innovations (reckoned as experiment) were 22.2%, almost as much as theory.

The Higgs mechanism, theoretically predicted in 1962, eventually culminated in the announcement of the experimental discovery of the Higgs boson in 2012 at the Large Hadron Collider (LHC). The LHC employs thousands of scientists and engineers. F. Englert and P.W. Higgs received the 2013 Nobel Prize in Physics for their theoretical work. The Higgs Nobel Prize was a rare instance where theory preceded experiment. Einstein, like most theorists to win, won the prize for explaining an experiment.

Einstein's theory of relativity is arguably of the most precisely tested theory in science, with experimental agreement to better than the 12th decimal place. The measurements allowing such fine precision and accuracy involved a binary pulsar (Antoniadis et al. 2013). Although it is so well tested, several times a decade we read about an experiment claiming to violate or refute Einstein's theory. The experimental results which appear to refute are expertly considered and critiqued. Invariably a flaw in technique or instrumentation is discovered, further confirming Einstein's theory rather than refuting it. Confidence in Einstein's theory increases with each test. In 2018 a test beyond our galaxy was reported and confirmed.

We trust Einstein's theory as far as it has been experimentally tested, not due to its popularity.

In the early 1980s, String Theory became a popular physics field. Our particular interest for this text is twofold: (i) its fashionability invites comparison with Ioannidis corollaries 5 and 6; (ii) Ioannidis evaluated medical research based on falsifiable predictions, called “PPV,” as discussed in Panel 2.1.

During the peak years of String Theory popularity, its math techniques flourished. It garnered the majority of physics funding; its proponents placed the majority of professorships. We avidly read about it.

Since String Theory is one of several competing theories, and outside our specialty, we continue to watch with interest all sides in the dispute.

String Theory notably depended on multiple spatial dimensions. E.A. Abbott's book Flatland: A Romance of Many Dimensions had already introduced us to imagining extra spatial dimensions. E. Witten was a top advocate of String Theory. A principal concern, stated by advocates and critics alike, was that String Theory lacked predictions that could be tested experimentally.

Beginning in 2006, the warnings in Ioannidis corollaries 5 and 6 compared with String Theory. Theoretical physicist L. Smolin raised an alert in his book The Trouble with Physics: The Rise of String Theory, the Fall of a Science and What Comes Next (2006). Noting that physics was rich in alternative “promising new directions,” Smolin wrote to promote other areas of theoretical and experimental physics. Others noticed as well. P. Woit wrote his critique Not Even Wrong: The Failure of String Theory and the Continuing Challenge to Unify the Laws of Physics (2006).

Roger Penrose 6 took on three popular areas of physics, expressing similar concerns. Penrose's recent book (2016) is Fashion, Faith, and Fantasy in the New Physics of the Universe . The chapter entitled “Fashion” covers String Theory, adding his perspective. The chapter covering quantum mechanics is entitled “Faith.” Techniques of quantum electrodynamics show success, the evidence being experimental validation to accuracies rivaling relativity. Penrose added an experimental test of macro‐quantum superposition. The chapter entitled “Fantasy” deals with cosmologies beyond the big bang. Penrose highlighted “The Phenomenal Precision in the Big Bang”; of particular interest to us in thermo‐fluids was the necessarily low entropy.

Regardless of a theory's popularity, political or otherwise, we urge experimental tests.

Sometimes an experiment planned for another purpose provides the answer. A prime example was the search for residual thermal evidence of the big bang. While a Princeton physics group was proposing to test the theory, a couple of astronomers at AT&T Bell Labs, Arno Penzias and Robert Wilson, were trying to eliminate noise which was contaminating the signal in their antenna. They even removed pigeon and bat residue. Failing to eliminate the noise, they spoke with Princeton professor Robert Dicke. They published “A Measurement of Excess Antenna Temperature at 4080 Mc/s” in the Astrophysical Journal (Penzias and Wilson 1965). In 1978, Penzias and Wilson won the Nobel Prize.