Robert J. Moffat - Planning and Executing Credible Experiments

1 3.1 From your review of prior work and research, what risks might your experiment encounter?

2 3.2 What risks will your client accept?

3 3.3 What risks will your client face by not pursuing this experiment?

4 3.4 How much will your client stake on successful completion of this experiment?

5 3.5 What is the cost of a null answer?

6 3.6 How much risk will your client assume?

7 3.7 Do the Occupational Safety and Health Administration (OSHA) or state laws or national codes limit the direction and extent that your proposed experiment can pursue?

8 3.8 Rinse and repeat from Exercise 3.1.

The motivating question is the question that, if answered, justifies the entire cost of running the experiment.

I strongly believe in organizing every research‐type experiment around a question, for several reasons.

1 When your goal is to answer a question, you know you can quit when you have an acceptable answer!

2 If your experimental objective was “to study…,” or “to investigate…,” or “to document…,” then you may never know when to quit. There is no end to “studying” or “investigating.” You will quit only when the money runs out or you get bored or reassigned, but you will never be finished. If, on the other hand, you have a specific question to answer, you can quit when you have the answer or when you can prove that you can't get the answer by this kind of experiment.

3 Knowing the motivating question helps in making the trade‐off decisions during planning and debugging.

A research program generally begins with a “need to know,” an urge on someone's part to solve a problem. Unfortunately, the person who first feels the urge may have in mind some steps toward what he/she thinks is the solution and may present an experiment plan that is a path to that particular “solution” rather than a path to solving the general problem.

To some extent, this can be avoided by specifically addressing the issue of “What question are we trying to answer?” instead of “What are we going to do?”

I (RM) came to this approach to experiment planning after many years of dealing with talented graduate students, each eager to get on with their programs but frequently stalling out when obstacles arose in the lab. I would find them at my door wondering what to do about the latest nuisance. Trying the simple approach of answering their questions, I found that the pace of research was then dictated by the number of hours I spent dealing with their problems. It was like pushing a rope! Every obstacle would bring the program to a halt. This sort of experience must be more common than I had thought, because it is the focus of an American folk song, “There's a Hole in My Bucket,” which is presented below for your amusement ( Panel 4.1).

With the help of (and some resistance from) my students, I discovered the problem: I was not communicating well enough. The students and I did not always share the same picture of what should be done next. I decided that this was because I was trying to tell them what to do, whereas I should have been telling them where they were trying to go. I began to address the central issue: how could the student and I come to a meeting of the minds such that they would be able to solve all of their problems without coming to me?

Ever been through one of these discussions? I have, many of them – with grad students who didn't have a clear picture of what needed to be done!

This led to the practice of formulating the lab programs in terms of the questions we were trying to answer.

I view the motivating question as a sort of intellectual nail driven into the wall of the world and to which the student is tied by some sort of intellectual “rope.” Once the motivating question has been really accepted (that is, thoroughly understood and accepted), the student is “tied” to that nail by her/his own rope of intent to finish. When students encounter an obstacle, they can pull themselves past it, always choosing a solution that moves them closer to the objective. This is a far different dynamic than the student trying to respond to my suggestions of what to do next. Once this approach was in place, I no longer felt I was trying to push ropes. 1

I strongly recommend figuring out the “motivating question” for each experiment as a way of focusing attention on what must really be done. There are other advantages that accrue from working to answer a question as opposed to working to take data or study something. I hope they will become clear in this chapter.

How can one identify the motivating question that drives an experiment, and who has the right to make this identification? Not the experimenter! Rather, it is the client who has this right because this is the person who is paying the bill for the experiment!

Identification of the motivating question is rarely easy. The urge to learn something may be strongly felt, but expressing exactly what you want to learn is very difficult. It is bad enough trying to talk about the subject when you have an hour or more to try to make clear your intentions. It is incredibly difficult to write down a concise description in such language that it cannot be misunderstood – and that is what we are trying to do. How often has the exercise of writing exposed our own sloppy thinking? How often have we felt despair at “trying to get it right” in writing? Take courage! The end result, your motivating question, is worth your effort to identify it early!