Robert J. Moffat - Planning and Executing Credible Experiments

Platt's contribution is his reminder to seek disproof for hypotheses, rather than support, as a more economic strategy.

Once the question has been determined, the next step is to agree on the form of an acceptable answer. A clear agreement on the form of an acceptable answer helps ensure that both parties have the same question in mind.

Thinking about the form of an acceptable answer may, in fact, lead to refining the question, especially if you play “devil's advocate” and look for “silly” answers. If “silly” answers are not precluded by the form of the question, then the question needs to be refined.

The question “What is the effectiveness of this heat exchanger at its design conditions?” needs only a single number for an answer: “The effectiveness is 0.86.”

Another example, using instrument calibration, has more latitude. If the calibration question was: “Does this instrument meet its accuracy specification?” then a satisfactory answer would be a simple “Yes” or “No.” If, however, the question was “How do the readings of this instrument compare with the true values, over its range?” then an acceptable answer would require a table of values, or a chart, listing the indicated value corresponding to each true value.

Note that a simple request to “Calibrate this instrument” does not establish which of these two answers would be acceptable and does not qualify as a desirable motivating question.

The third point that should be dealt with is uncertainty. How much uncertainty can be tolerated in the answer to the motivating question?

For most heat‐transfer situations, an uncertainty of ±5% is “Olympic‐quality” data. Much of our present heat‐transfer understanding was developed from historic and legacy databases with ±20% uncertainty. Often, the client already has some data or has a design approach that works reasonably well but wants to reduce the uncertainty. In such a situation, to produce more data with the same uncertainty as the existing dataset will not advance the art, nor will it satisfy the client.

The allowable uncertainty should be described in a manner consistent with current conventions. For most engineering work, the uncertainty is specified at the 95% confidence level and presumes a symmetric probability distribution, centered around zero error. For example, if the uncertainty is quoted as ±10% at the 95% confidence level, this means that 19 of 20 repeated trials of the same measurement act will yield a value within ±10% of the quoted value. The possible error is, by agreement, equally likely to be positive or negative.

In some cases, the client may wish to specify a “one‐sided” tolerance, for example to demonstrate that a product is better than its competition. A one‐sided test must be selected before the experiment is run. The one‐sided test must be clearly indicated in the report so that the results can be interpreted accurately in terms of the one‐sided error allowance. Otherwise, the experiment will be reckoned as a two‐sided or “zero‐centered” test.

From my experience, a written memo of understanding should be prepared that clearly states the motivating question, the form of an allowable answer, and the acceptable uncertainty. This should become a part of the experimental record.

It is difficult for some project managers to sign off on such a statement as, “The answer to this question, with no other benefit, will justify the cost of this experiment.” Lurking in the back of most minds are hopes for “spin‐off data,” bits of knowledge they hope to get, or a general feeling that the experiment ought to be “kept loose, so we can steer it as we go.” These urges usually arise out of reluctance to come to grips with the specific problem: what is it that we really must know?

I recommend organizing a “signing party” at which the people with a stake in the issue sign off on the question, the form of its answer, and the uncertainty. Too often, higher management pays little or no attention to these “philosophical issues” until too late. There is something about deliberately signing off that makes people pay attention.

Unless there is explicit agreement, early, on the objective of the experiment, a host of serious issues may ensue:

Disagreements on “what to do next.”

Dissatisfaction with the outcome.

“Second guessing” after the final report.

A general whitewashing to cover up the lack of definite progress.

Too many experiments simply don’t answer any question – they only contribute some facts that may or may not be useful.

1 Platt, J.R. (1953). Address before the Division of Physical Chemistry. American Chemical Society.

1 4.1 Propose at least three potential motivating questions that your experiment can answer.

2 4.2 Discuss with your client each of the potential motivating questions. Which one is preferred?

3 4.3 What questions does your client raise in light of the preferred motivating question?

4 4.4 How can this motivating question be refined? Can it be made more concrete?

5 4.5 Can your motivating question be clearly achieved?

6 4.6 Rinse and repeat from Exercise 4.1.

1 1Upon my (Moffat) retirement from active teaching, my recent graduates presented me with a memorial plaque bearing a gold‐plated piece of rope and the legend “You Can't Push a Rope.” This saying is one of the first major lessons every engineer learns in the course Statics, when she/he draws forces on an object.

2 2On an exam in a graduate heat‐transfer course, I once asked: “When can equation 8‐16 be used?” One student responded, “On Thursday!” What could I say!

3 3In one program, it took nearly a year to get the final form of the question worked out. The rig was nearly ready for shakedown runs before we were content. That question, written on a 3 × 5 card and thumbtacked to the wall in the office, guided our research through three PhD theses.

Текст предоставлен ООО «ЛитРес».

Прочитайте эту книгу целиком, купив полную легальную версию на ЛитРес.

Безопасно оплатить книгу можно банковской картой Visa, MasterCard, Maestro, со счета мобильного телефона, с платежного терминала, в салоне МТС или Связной, через PayPal, WebMoney, Яндекс.Деньги, QIWI Кошелек, бонусными картами или другим удобным Вам способом.