John Paulos: INNUMERACY: Mathematical Illiteracy and Its Consequences

One more example of a statistical test. Suppose I hypothesize that at least 15 percent of the cars in a certain region are Corvettes, and upon watching one thousand cars go by representative intersections in the region note only eighty Corvettes among them. Using probability theory, I calculate that, given my assumption, the likelihood of this result is well below 5 percent, a commonly used "level of significance." Therefore I reject my hypothesis that 15 percent of the cars in the region are Corvettes.

There are two sorts of errors that can be made in applying this or any statistical test; they're called, imaginatively enough, Type I and Type II errors. A Type I error occurs when a true hypothesis is rejected, and a Type II error occurs when a false hypothesis is accepted. Thus, if a large number of Corvettes from a car show drove through the region and we therefore accepted the false assumption that at least 15 percent of the cars in the region were Corvettes, we would be making a Type II error. On the other hand, if we didn't realize that most of the Corvettes in the region weren't driven but were kept in garages, then in rejecting the true assumption we would be making a Type I error.

The distinction can also be applied less formally. When money is being distributed, the stereotypical liberal tries especially hard to avoid Type I errors (the deserving not receiving their share), whereas the stereotypical conservative is more concerned with avoiding Type II errors (the undeserving receiving more than their share). When punishment is being meted out, the stereotypical conservative is more concerned with avoiding Type I errors (the deserving or guilty not receiving their due), whereas the stereotypical liberal worries more about avoiding Type II errors (the undeserving or innocent receiving undue punishment).

Of course, there are always people who will object to the strictness of the Food and Drug Administration in not releasing drug X soon enough to prevent suffering, and also complain loudly when drug Y is released prematurely and causes severe complications. Like the FDA, which must evaluate the relative probabilities of a Type II error (okaying a bad drug) and a Type I error (not okaying a good drug), we must constantly evaluate analogous probabilities for ourselves. Should we sell the rising stock option and risk losing out on its further ascent, or hold on to it and risk its decline and the loss of our premium? Should we operate, or manage medically? Should Henry ask Myrtle out and risk her saying no, or should he not and keep his peace of mind but not learn that she would have said yes?

Similar considerations apply to the manufacturing process. Often, after some crucial bit of machinery breaks down because of bad parts, or after some unusually unreliable string of items (firecrackers, cans of soup, computer chips, condoms) comes to light, there are calls for new controls to ensure that no more defectives are manufactured. This sounds reasonable, but in most cases it's simply impossible or, what amounts to the same thing, prohibitively expensive. There are quality-control checks whereby samples of each batch of manufactured goods are tested to ensure that there are no or very few defectives in the sample, but not every item is tested (or even testable).

There's almost always a trade-off between quality and price, between Type II errors (accepting a sample with too many defectives) and Type I errors (rejecting a sample with very few defectives). Moreover, if this trade-off is not acknowledged, there is a tendency to deny or cover up the inevitable defective items, which makes the job of quality control that much more difficult. Apropos of this is the proposed Strategic Defense Initiative, whose computer software, satellites, mirrors, etc., would be so awesomely complex that it takes a kind of innumerate naivete to believe it will work without bankrupting the treasury.

The Strategic Defense Initiative brings thoughts of destruction and salvation, and even here the notion of trade-offs can play a useful role. Pascal's wager on God's existence, for example, can be cast as a choice between the relative probabilities and consequences of Type I and Type II errors. Should we accept God and act accordingly and risk a Type II error (He doesn't exist), or should we reject God and act accordingly and risk a Type I error (He does exist). Of course, there are assumptions underlying these phrases which are invalid or meaningless without clarification. Still, the point is that all kinds of decisions can be cast into this framework and call for the informal evaluation of probabilities. There is no such thing as a free lunch, and even if there were, there'd be no guarantee against indigestion.

Estimating characteristics of a population, such as the percentage who favor a certain candidate or a particular brand of dog food, is, like hypothesis testing, simple in principle. One selects a random sample (easier said than done) and then determines what percentage of the sample favors the candidate (say, 45 percent) or the brand of dog food (say, 28 percent), which percentages are then taken to be estimates of the opinion of the population as a whole.

The only actual poll I ever took myself was informal and designed to answer the burning question: What percentage of college women enjoy watching the Three Stooges? Eliminating those unfamiliar with the Stooges' slapstick, physical, lowbrow comedy, I found that an overwhelming 8 percent of my sample admitted to such an indulgence.

The care devoted to the selection of the above sample was not great, but at least the result, 8 percent, had a credible ring to it. One obvious problem with statements such as "67 percent (or 75 percent) of those surveyed favored tablet X" is that they may be based on tiny samples of three or four. Even more extreme is the case where a celebrity endorses a diet or medicine or whatever, in which case we have a sample of one, and generally a paid sample at that.

Thus, more difficult than making statistical estimates is deciding how much confidence we should place in them. If the sample is large, we can have more confidence that its characteristics are close to those of the population as a whole. If the distribution of the population is not too dispersed or varied, we can, again, have more confidence that the sample's characteristics are representative.

By utilizing a few principles and theorems in probability and statistics, we can come up with so-called confidence intervals to estimate how likely a sample characteristic is to be representative of the population as a whole. Thus, we might say that a 95 percent confidence interval for the percentage of voters favoring candidate X is 45 percent plus or minus 6 percent. This means that we can be 95 percent certain that the population percentage is within 6 percent of the sample percentage; in this case, between 39 percent and 51 percent of the population favor candidate X. Or we might say that a 99 percent confidence interval for the percentage of consumers preferring brand Y dog food is 28 percent plus or minus 11 percent, meaning that we can be 99 percent certain that the population percentage is within 11 percent of the sample percentage; in this case, between 17 percent and 39 percent of consumers prefer brand Y.

As with the case of hypothesis testing, however, there is no free lunch. For samples of a given size, the narrower the confidence interval-that is, the more precise the estimate-the less confident we can be of it. Conversely, the wider the confidence interval-that is, the less precise the estimate-the more confident we can be of it. Of course, if we increase the size of the sample, we can both narrow our interval and increase our confidence that it contains the population percentage (or whatever the characteristic or parameter is), but it costs money to increase sample sizes.