Robert J. Moffat - Planning and Executing Credible Experiments

Figure 2.1 Rice cooker design trajectory.

How amazing is the human sensory system and what it enables us to observe! Trying to extract as much information using scalar measuring instruments is quite a challenge. The human sensory system is very complex, and its receptors are very well tuned to our environment. If our eyes were just a few decibels more sensitive, we could see single photons; if our ears were just a few decibels more sensitive, we could hear the Brownian motion of individual air molecules as they bounced off our eardrums.

Consider our sense of touch. Our machinists claim to easily detect surface roughness of 50 mils (1 μ m) with work‐calloused hands. Can our body measure force? Can it measure temperature? We sense not temperature directly but heat‐transfer rate; if you've ever dipped a cold toe into a warm bath, did the water feel boiling hot even if was just warm? (As in an Onsen hot spring in Japan.) We don’t feel force directly but pressure and shear force. In contrast, in the lab we have simple instruments for measuring force and temperature but need sophisticated techniques to measure heat‐transfer rate. Even touch is a marvel. Prepublication we learned: in the journal Nature , it has been “experimentally established that humans have the capacity to perceive single photons of light” (Tinsley et al. 2016). Furthermore “human tactile discrimination extends to the nanoscale … within billionths of a meter” (Skedung et al. 2013). In part this explains how polished steel tactilely differs from smooth rubber.

One of the first problems to be faced in exploratory research, and in development work, is identifying which scalars are significant to the issue at hand. Sometimes a single scalar is sufficient, such as a temperature, pressure, or velocity. Sometimes a compound scalar measure can be put together from a set of simple scalars. One example would be the Reynolds number, used for characterizing the state of the flow in a channel. Sometimes two or more scalars can be combined into one measure to reflect value judgments or “trade‐offs” in desirability. For example, consider the problem of selecting the optimum heat exchanger for an engine application. In even the simplest situation, ignoring such considerations as cost, size, and durability, a heat exchanger for engine service has at least two important scalar descriptors: its effectiveness and its pressure drop. Typically, high effectiveness is “good,” while high‐pressure drop is “bad.” Also typically, pressure drop goes up when effectiveness goes up. Neither by itself is a measure of goodness for the heat exchanger. A weighted sum of the two can be used as a “composite scalar” by finding two weight functions, one for effectiveness and one for pressure drop, which accounts for the effects of both on some single, important parameter – brake‐specific fuel consumption is another example. Such “goodness factors” can be used to account for many factors at a time and to convert a nonmeasurable situation to a measurable one.

The choice of what to measure sets the course of the entire experiment, and that choice should be made with considerable care.

Always remember as you plan: making a measurement intrudes on the system measured. Ask: How much does the measurement intrusion alter the system? How does measurement affect the accuracy and precision of its results? Furthermore, the intrusion affects the uncertainty of the measurement. A prime motivator for inventing new probes and experimental techniques is the aim to intrude less when taking a measurement. Classic measurement is the focus of this text.

In certain quantum physics experiments, however, mere observation alters the system. In these experiments, if an observation is made the system has one result; if no observation is made the system shows contrary results. These tests have been reproduced worldwide. If you are interested, we recommend Richard Muller's book (Muller 2016). Or search “theory of measurement in physics.”

There is an unfortunate tendency, among engineers particularly, simply to measure everything which can be measured, report the results, and hope that someone, someday, will find the results useful. This is a deplorable state of affairs, even though it has a long and honored past. In many respects, it follows the scientific tradition which emerged from Europe in the nineteenth century, when the art of measurement expanded so rapidly in Western civilization.

William Thomson, Lord Kelvin, famously proclaimed: “When you can measure what you are talking about and express it in numbers then you have the beginning of knowledge.” That is still true today but with some limitations (Thomson 1883).

Thomson's enthusiasm for measurements should be interpreted in terms of the times in which he lived. The European scientists of that period were infatuated with measurement. Every new measurement technique developed was applied to every situation for which it seemed to fit. There was no storehouse of knowledge about the physical world. Each new series of measurements revealed order in another part of the physical world, and it appeared that every measurement answered some question, and every question could be answered if only enough measurements were made. That was true partly because there were so many unanswered questions and partly because most of the questions which were being asked at that time could be answered by scalars.

Since that era, experimental work has become more expensive and more complex. The questions that lead us into the lab today usually involve the behavior of systems with many components or processes with several simultaneous mechanisms. It is not easy to translate a “need to know” into an experiment under such complex conditions. The first problem is deciding what scalars (i.e. what measurable items) are important to the phenomenon being investigated. This step often takes place so fast and so early in a test program that its significance is overlooked. When you choose what to measure, you implicitly determine the relevance of the results.

The early years of the automobile industry provide at least one good example of the consequences of “leaping in” to measurements. As more and more vehicles took to the road, it became apparent that some lubricating oils were “better” than others, meaning that automobile engines ran longer or performed better when lubricated with those oils. No one knew which attributes of the oils were important and which were not, and so all the easily measured properties of the “good” oils were measured and tabulated. The result was a “profile of a good oil.” The oil companies then began trying to develop improved oils by tailoring their properties to match those of the “good oil profile.” The result was a large number of oils that had all the desired properties of a good oil except one: they didn't run well in engines! 4

The techniques of experimentation differ considerably from the techniques of analysis, by the nature of the two approaches. It is worthwhile to examine some of these differences.

Analysis is deductive and deals with the manipulation of a model function. The typical problem is: Given a set of postulates, what is the outcome? Experiment is inductive and deals with the construction of models. The typical problem is: given a set of input data and the corresponding output data, what is the form of the model function that connects the output to the input? The analytical techniques are manipulative, whereas the experimental techniques are those of measurement and inference.