John Paulos - INNUMERACY - Mathematical Illiteracy and Its Consequences

Surveys or polls which don't include confidence intervals or margins of error are often misleading. More often than not, surveys do include such confidence intervals, but they don't make it into the news story. Hedging or uncertainty is rarely newsworthy.

If the headline reads that unemployment declined from 7.1 percent to 6.8 percent and doesn't say that the confidence interval is plus or minus 1 percent, one might get the mistaken impression that something good happened. Given the sampling error, however, the "decline" may be nonexistent, or there may even be an increase. If margins of error aren't given, a good rule of thumb is that a random sample of one thousand or more gives an interval sufficiently narrow for most purposes, while a random sample of one hundred or less gives too wide a margin for most purposes.

Many people are surprised at how few individuals pollsters survey to get their results. (The width of the confidence interval for percentages varies inversely as the square root of the size of the sample.) Actually, they generally poll a larger number than is theoretically necessary to compensate for problems associated with getting a random sample. When the random sample selected contains one thousand people, the theoretical 95 percent confidence interval for estimating the percentage who favor candidate X or dog food Y is about plus or minus 3 percent. Pollsters often use plus or minus 4 percent for this sample size because of nonrespondents and other difficulties.

Consider the problems associated with a typical telephone poll. Will the results be affected by leaving out homes without a telephone? What percentage of people refuse to respond, or hang up when they learn a pollster is calling? Since the numbers are chosen at random, what's done when a business phone is reached? What if no one is home, or a child answers the phone? What effect does the sex (or voice or manner) of the telephone interviewer have on the responses? Is the interviewer always careful or even honest in recording the responses? Is the method for choosing exchanges and numbers random? Are the questions leading or biased? Are they comprehensible? Whose answer counts if there are two or more adults at home? What methods are used to weigh the results? If the poll concerns an issue about which opinions are changing rapidly, how are the results affected by spreading the poll out over time?

Similar difficulties apply to personal-interview polls and mail polls as well. Asking leading questions or using an insinuating tone is a common pitfall of personal-interview polls, while an especially important concern in mail polls is avoiding self-selected samples where the most committed, aroused, or otherwise atypical groups of people are more likely to be respondents. (Such self-selected samples sometimes go by the more honest term of "lobby.") The famous 1936 Literary Digest poll which predicted Alf Landon would beat Franklin Roosevelt by a three-to-two margin was wrong because only 23 percent of the people who were sent questionnaires returned them, and these generally were wealthier. A similar shortcoming biased the 1948 poll which showed Thomas Dewey beating Harry Truman.

Magazines and newspapers are notorious for announcing biased results based on responses to questionnaires appearing in the periodical. These informal polls rarely come with confidence intervals or any details of the methods used, so the problem of self-selected samples is not always immediately apparent. When feminist author Shere Hite or columnist Ann Landers reports that a surprisingly high percentage of their respondents are having affairs or would rather not have had children, we should automatically ask ourselves who is most likely to answer these questionnaires: someone having an affair or someone reasonably content, someone exasperated by her kids or someone happy with them.

Self-selected samples are not much more informative than a list of correct predictions by a psychic. Unless you get the complete list of predictions or a randomly selected subset, the correct predictions mean nothing. Some of them are bound to turn out true by chance. Similarly, unless your poll sample is randomly selected and not self-selected, the poll results usually mean very little.

In addition to being wise to the problem of self-selected samples, the numerate consumer should also understand the related problem of the self-selected study. If company Y commissions eight studies comparing the relative merits of its product and that of its competitor, and seven of the eight conclude that its competitor's product is superior, it's not hard to predict which study company Y will cite in its television commercials.

As in the chapters on coincidence and pseudo-science, we see that the desire to filter and emphasize information is at odds with the desire to obtain a random sample. Especially for the innumerate, a few vivid predictions or coincidences often carry more weight than much more conclusive but less striking statistical evidence.

Because of this, it's unclear to me why a collection of intimate profiles or personal stories is so frequently termed a poll. If done well, such a collection is more engaging (even if less convincing) than the typical poll or survey and loses much of its value when wrapped in the ill-fitting shroud of a scientific survey.

The name of the game in statistics is the inferring of information about a large population by examining characteristics of a small, randomly selected sample. The techniques involved-from the enumerative induction of Francis Bacon to the theories of hypothesis testing and experimental design of Karl Pearson and R. A. Fisher, the founding fathers of modern statistics-all depend on this (now) obvious insight. Several unusual ways of obtaining information follow.

The first, which will perhaps become increasingly important in an inquisitive age which professes to still value privacy, makes it possible to obtain sensitive information about a group of people without compromising any person's privacy. Assume we have a large group of people and want to discover what percentage of them have engaged in a certain sex act, in order to determine what practices are most likely to lead to AIDS.

What can we do? We ask everyone to take a coin from his or her purse or wallet and direct them to flip it once. Without letting anyone else see the outcome, they should note whether it lands on heads or tails. If the coin lands heads, the person should answer the question honestly: Has he or she ever engaged in the given sexual practice-yes or no? If it comes up tails, the person should simply answer yes.

Thus, a yes response could mean one of two things, one quite innocuous (the coin's landing tails), the other potentially embarrassing (engaging in the sex act). Since the experimenter can't know what yes means, people presumably will be honest.

Let's say that 620 of 1,000 responses are yes. What does this indicate about the percentage of people who engage in the sex act? Approximately 500 of the 1,000 people will answer yes simply because the coin landed tails. That leaves 120 people who answered yes out of the 500 who replied to the question honestly (those whose coins landed heads). Thus, 24 percent (120/500) is the estimate for the percentage of people who engage in the sex act.

There are many refinements of this method that can be used to learn more detail, such as how many times people engaged in the sex act. Some variations of the method can be more informally implemented, and could be used by a spy agency to estimate the number of dissidents in an area, or by an advertising agency to estimate the market for a product whose attractiveness people are likely to deny. The raw data for the calculations can come from public sources and, appropriately massaged, can yield surprising conclusions.