Robert J. Moffat - Planning and Executing Credible Experiments

This chapter addresses how to formulate the motivating question and the advantage of working to a question you are trying to answer. If you experiment without a motivating question, you will take a series of steps – but when will you arrive?

When the urge strikes to run a new experiment, don't fight it. If the urge has struck you, then talk to yourself – takes notes on what you think you should do, what you think the rig might look like, or what you think might happen. Let this process run until you feel you have really expressed yourself. Then stop and ask, “What question am I trying to answer?”

If the urge has struck someone else, and you are that person’s sounding board, listen actively and take notes. Let the person keep talking until he or she runs dry – then the individual knows you have really heard the idea, entirely. Then ask, “If you do this, and you are successful, what question will you have answered?”

I think you must honor the enthusiastic urge by letting it have center stage until it plays itself out. I simply ask that the speaker (myself or my student) get reasonably specific right away. This is not intended in a critical sense but simply to detail the plan. I take notes about the proposed apparatus, the test conditions, and the proposed data, trying to absorb the real intent of the idea. I keep at this until the speaker (myself or the student) runs out of things to say about what to do.

At this point everything that could be said has been said. The pressure is off. We can get to work.

Now I raise the following five questions.

If we do this experiment and get all the data we have asked for:

Q1. What question will we have answered?

Q2. Is this question worth the cost of answering it?

Q3. Has it already been answered?

Q4. Is the proposed experiment the best way to answer it?

Q5. If we get what we asked for, will that solve the problem that led to this work?

Almost always, trying to answer Q1 honestly and carefully puts the issue in a new light.

Once attention is focused on what we want to accomplish instead of on what we want to do , we can admit the possible existence of other ways to accomplish the same objective. This often leads us to formulate a more important question than we first had in mind and to propose a quite different experiment.

It is a good idea to start by formulating several versions of what seems to be the motivating question, and then start critiquing them. The first versions will leave a lot unsaid, and a good “devil's advocate” approach will make some of them look downright naive. 2 Don't be reluctant to play devil's advocate – if the question allows a silly answer, the question is poorly formulated. Precision will soon start to emerge.

When you have a few good candidates, the test for identifying a really good motivating question is, “Which of these questions, if it alone is answered, will justify the cost of this experiment?”

There can be only one “top priority question.” During the planning of the experiment, when a trade‐off is required, the motivating question provides the decision criterion. The motivating question is the “mission statement” of the experiment.

Sometimes, the question keeps on being refined, even while the program is in motion. 3 No harm in that.

In terms of experiment planning, it helps to distinguish three levels of questions that might be raised during the development of an experiment plan:

Level 1: What happens?

Level 2: By what means does it happen?

Level 3: What are the underlying physics?

It is important to identify the level of the motivating question and ensure that all following decisions are aimed at progress at that same level. The different levels lead to different experiments for a given situation.

For example, consider a program aimed at studying the effects of high free‐stream turbulence u ' on the value of the heat‐transfer coefficient h on a flat plate.

The Level 1 question would be: “How does the heat‐transfer coefficient h vary with free‐stream turbulence u ', all other factors remaining fixed?” This question would be answered by a series of experiments in which h was measured for different values of u ' for some specified set of conditions.

A Level 2 question would be: “What changes in the structure of the boundary layer were responsible for the changes in h ?” This question would be answered by measurements of the velocity and temperature distribution within the boundary layer, seeking a correlation between high h and some recognizable change in the boundary layer structure.

A Level 3 question would be: “How have the momentum and energy transports within the boundary layer been altered by the turbulence?” This question would be answered by measurements of the mixing length and turbulent Prandtl number or the turbulence intensity and dissipation length scale within the boundary layer.

Note that these different levels require entirely different types of measurements, so it is important to know which level is most important to the client, the ultimate customer.

There is an implicit presumption in modern fluid mechanics and heat transfer that low‐order events can be predicted from high order, i.e. that knowledge of the turbulence transport mechanisms will allow calculation of the boundary layer structure, and that knowledge of the boundary layer structure will allow prediction of h at the surface.

This hierarchical assumption leads some experimenters to immediately start work at Level 2 or Level 3, seeking an elegant and powerful answer to the Level 1 question. In general, however, it seems to work best if one directly addresses the Level 1 question first over the entire range of the desired conditions before attempting to shift up to Level 2 and Level 3.

It is particularly important not to inadvertently mix levels in the same experiment plan. One may pursue two, or even three, levels of questions within one series of tests, and that is frequently done to save testing time, but one should remain aware that each level requires different data and that the different datasets are addressing separate objectives.

A particularly powerful approach to selection of the “right” question is described by John R. Platt, professor of physics and biophysics at the University of Chicago (1953), as “strong inference.” The essential features of strong inference are (1) the formulation of more than one alternative hypothesis concerning the major question, and (2) the execution of a set of experiments that have the possibility of disproving each hypothesis. The power of the method lies in the fact that, once a hypothesis has been disproved, then no further work need be done along that line. Note that if an experiment simply supports a hypothesis, not much progress has really been made, because the very next experiment may disprove it. To use strong inference, one must assemble systems of hypotheses that fully enclose the main question so that at least one hypothesis must finally survive its experimental test.

Platt lists the following steps:

1 Devise alternative hypotheses.

2 Devise a crucial experiment (or several of them) with alternative possible outcomes, each of which will as nearly as possible exclude one or more of the hypotheses.

3 Carry out the experiments so as to get clean results.

4 Recycle the procedure, making subhypotheses or sequential hypotheses to refine the possibilities.

Identification of the motivating question corresponds to the step devising alternative hypotheses. The specific experimental objective is, then, to test these hypotheses by experiment.