Robert J. Moffat - Planning and Executing Credible Experiments

4 4Dr. Loyd Withrow, GM Research Labs, ca. 1953. Personal communication.

5 5Thank you to Dr. Ioannidis for permission to include the lists from his articles.

6 6Penrose trained Stephen Hawking; together they wrote a number of landmark articles on the nature of the universe. Penrose's book Road to Reality (2005) highlights the math that underlies various areas of physics. His presentation of the reality and necessity of complex numbers is the best I've (RH) read.

My background is mainly in research and development experiments in heat transfer and fluid physics. When I think of planning an experiment, I think about wind tunnels, heat exchangers, temperature, and flow control. I have tried to generalize my experience, but my background certainly colors my outlook.

Experimental work is expensive. Although costs vary for different fields and situations, the time costs are all similar: all laboratory work runs in real time, an hour for an hour, and there are no short cuts. It is important that experiments be well conceived, well executed, and well documented.

The purpose of experiments is to produce provably accurate data that answer an agreed‐upon question about the behavior of a system. This is the kind of experiment I wish to discuss. Some of the important ideas will be transferable to other types of experiment, but the main thrust here is to deal with R&D experiments with tangible, numerical objectives.

The planning of such an experiment is necessarily iterative. The process starts with a tentative plan: a first impression of the goal, a plausible experimental approach, a possible suite of instruments, a tentative set of tests to run. This plan must then be challenged: will it produce the desired information with acceptable accuracy? Then the plan is refined, sometimes by improving the statement of objectives, sometimes by selecting a better approach, sometimes by improving the instrumentation.

The general steps in an experimental program can be summarized in the following outline:

1 Assignment.

The Iterative Loop

1 Determine the objectives.

2 Select the experimental approach.

3 Parametrically design an apparatus.

4 Design apparatus hardware.

5 Construct and install apparatus.

6 Design analysis software; debug with fabricated sample data.

7 Perform shakedown, debugging, and qualification runs.

The Execution

1 Collect data.

2 Reduce data and analyze.

3 Report.

Seldom is a successful research experiment designed on a once‐through basis. This is not surprising when you think about the amount of scratch paper usually generated in trying to develop an original analysis. Alternative experimental approaches must be investigated and trade‐offs made between accuracy and convenience, range and speed, etc.

Research echoes sailing into uncharted territory. Sailboats cannot sail directly into the wind. The skipper must tack into the wind, repeatedly aiming right then left, iteratively correcting course.

This book takes an iterative approach to experiment planning. For example, it may not be clear how to execute some of the steps until later, when their background material has been developed. Some of the steps themselves may not even make sense until the background material has been developed. And some of the background material we ask for won't make sense until the need for it has been established. This circularity is typical of large‐scale projects with interactions: They cannot be studied sequentially, they have to be approached integrating “all at once” and iterating.

We emphasize the iterative nature of experiments. What seems plausible at first may not prove acceptable later. Sometimes a preliminary “exploratory” experiment is a good investment – to test an approach. It may show that the original concept of the experiment is not a good one. You likely will iterate the experiment itself, as in Figure 1.1. Details of the planning steps and their objectives are dealt with in subsequent chapters.

An additional issue of risk assessment warrants early treatment because it affects every decision in the planning process.

Consider the following risks:

The data may not answer the motivating question.

The data may not be provably accurate.

The schedule may not be met.

The budget may be exceeded.

Before the experiment is authorized to proceed, it must be established to the satisfaction of the reviewers that the motivating question can be answered with the desired accuracy, within the time allowed, and within the allowable budget.

Tables 3.1– 3.3elaborate the contents for the rest of this book – the topics that we feel should be addressed during the planning of an experiment.

Before some of the above topics can be addressed directly, some background material must be developed on the topology of experiments and the handling of experimental uncertainties.

Table 3.1 Overview of a research experiment plan.

Table 3.2Review the program plan. Do risk assessment and plan risk abatement. If satisfactory, go ahead. If not, go back.

Table 3.3 Assess the credibility of the program. Do risk assessment and plan risk abatement. If satisfactory, go ahead. If not, go back.