Richard Feynman - The Meaning of It All

The people who study medicine and try to cure people are interested in every method that they can find. And they have developed clinical techniques in which (all these problems are very difficult) they are trying all kinds of medicines too, and the woman got better. She also had chicken pox just before she got better. Has that got anything to do with it? So there is a definite clinical way to test what it is that might have something to do with it—by making comparisons and so forth. The problem is not to determine that something surprising happens. The problem is to make really good use of that to determine what to do next, because if it does turn out that it has something to do with the prayers of Mother Seaton, then it is worthwhile exhuming the body, which has been done, collecting the bones, touching many ribbons to the bones, so as to get secondary things to tie on other beds.

I now turn to another kind of principle or idea, and that is that there is no sense in calculating the probability or the chance that something happens after it happens. A lot of scientists don’t even appreciate this. In fact, the first time I got into an argument over this was when I was a graduate student at Princeton, and there was a guy in the psychology department who was running rat races. I mean, he has a T-shaped thing, and the rats go, and they go to the right, and the left, and so on. And it’s a general principle of psychologists that in these tests they arrange so that the odds that the things that happen happen by chance is small, in fact, less than one in twenty. That means that one in twenty of their laws is probably wrong. But the statistical ways of calculating the odds, like coin flipping if the rats were to go randomly right and left, are easy to work out. This man had designed an experiment which would show something which I do not remember, if the rats always went to the right, let’s say. I can’t remember exactly. He had to do a great number of tests, because, of course, they could go to the right accidentally, so to get it down to one in twenty by odds, he had to do a number of them. And its hard to do, and he did his number. Then he found that it didn’t work. They went to the right, and they went to the left, and so on. And then he noticed, most remarkably, that they alternated, first right, then left, then right, then left. And then he ran to me, and he said, “Calculate the probability for me that they should alternate, so that I can see if it is less than one in twenty.” I said, “It probably is less than one in twenty, but it doesn’t count.” He said, “Why?” I said, “Because it doesn’t make any sense to calculate after the event. You see, you found the peculiarity, and so you selected the peculiar case.”

For example, I had the most remarkable experience this evening. While coming in here, I saw license plate ANZ 912. Calculate for me, please, the odds that of all the license plates in the state of Washington I should happen to see ANZ 912. Well, it’s a ridiculous thing. And, in the same way, what he must do is this: The fact that the rat directions alternate suggests the possibility that rats alternate. If he wants to test this hypothesis, one in twenty, he cannot do it from the same data that gave him the clue. He must do another experiment all over again and then see if they alternate. He did, and it didn’t work.

Many people believe things from anecdotes in which there is only one case instead of a large number of cases. There are stories of different kinds of influences. Things that happened to people, and they all remember, and how do you explain that, they say. I can remember things in my life, too. And I give two examples of most remarkable experiences.

The first was when I was in a fraternity at M.I.T. I was upstairs typewriting a theme on something about philosophy. And I was completely engrossed, not thinking of anything but the theme, when all of a sudden in a most mysterious fashion, there swept through my mind the idea: my grandmother has died. Now, of course, I exaggerate slightly, as you should in all such stories. I just sort of half got the idea for a minute. It wasn’t something strong, but I exaggerate slightly. That’s important. Immediately after that the telephone rang downstairs. I remember this distinctly for the reason you will now hear. The man answered the telephone, and he called, “Hey, Pete!” My name isn’t Peter. It was for somebody else. My grandmother was perfectly healthy, and there’s nothing to it. Now what we have to do is to accumulate a large number of these in order to fight the few cases when it could happen. It could happen. It might have occurred. Its not impossible, and from then on am I supposed to believe in the miracle that I can tell when my grandmother is dying from something in my head? Another thing about these anecdotes is that all the conditions are not described. And for that reason I describe another, less happy, circumstance.

I met a girl at about thirteen or fourteen whom I loved very much, and we took about thirteen years to get married. It’s not my present wife, as you will see. And she got tuberculosis and had it, actually, for several years. And when she got tuberculosis I gave her a clock which had nice big numbers that turned over rather than ones with a dial, and she liked it. The day she got sick I gave it to her, and she kept it by the side of her bed for four, five, six years while she got sicker and sicker. And ultimately she died. She died at 9:22 in the evening. And the clock stopped at 9:22 in the evening and never went again. Fortunately, I noticed some part of the anecdote I have to tell you. After five years the clock gets kind of weak in the knees. Every once in a while I had to fix it, so the wheels were loose. And secondly, the nurse who had to write on the death certificate the time of death, because the light was low in the room, took the clock and turned it up a little bit to see the numbers a little bit better and put it down. If I hadn’t noticed that, again I would be in some trouble. So one must be very careful in such anecdotes to remember all the conditions, and even the ones that you don’t notice may be the explanation of the mystery.

So, in short, you can’t prove anything by one occurrence, or two occurrences, and so on. Everything has to be checked out very carefully. Otherwise you become one of these people who believe all kinds of crazy stuff and doesn’t understand the world they’re in. Nobody understands the world they’re in, but some people are better off at it than others.

The next kind of technique that’s involved is statistical sampling. I referred to that idea when I said they tried to arrange things so that they had one in twenty odds. The whole subject of statistical sampling is somewhat mathematical, and I won’t go into the details. The general idea is kind of obvious. If you want to know how many people are taller than six feet tall, then you just pick people out at random, and you see that maybe forty of them are more than six feet so you guess that maybe everybody is. Sounds stupid. Well, it is and it isn’t. If you pick the hundred out by seeing which ones come through a low door, you’re going to get it wrong. If you pick the hundred out by looking at your friends you’ll get it wrong because they’re all in one place in the country. But if you pick out a way that as far as anybody can figure out has no connection with their height at all, then if you find forty out of a hundred, then, in a hundred million there will be more or less forty million. How much more or how much less can be worked out quite accurately. In fact, it turns out that to be more or less correct to 1 percent, you have to have 10,000 samples. People don’t realize how difficult it is to get the accuracy high. For only 1 or 2 percent you need 10,000 tries.

The people who judge the value of advertising in television use this method. No, they think they use this method. It’s a very difficult thing to do, and the most difficult part of it is the choice of the samples. How they can arrange to have an average guy put into his house this gadget by which they remember which TV programs he’s looking at, or what kind of a guy an average guy is who will agree to be paid to write in a log, and how accurately he writes in the log what he’s listening to every fifteen minutes when a bell goes off, we don’t know. We have no right, therefore, to judge from the thousand, or 10,000, and that’s all it is, people who do this, who study what the average person is looking at, because there’s no question at all that the sample is off. This business of statistics is well known, and the problem of getting a good sample is a very serious one, and everybody knows about it, and it’s a scientifically OK business. Except if you don’t do it. The conclusion from all the researchers is that all people in the world are as dopey as can be, and the only way to tell them anything is to perpetually insult their intelligence. This conclusion may be correct. On the other hand, it may be false. And we are making a terrible mistake if it is false. It is, therefore, a matter of considerable responsibility to get straightened out on how to test whether or not people pay attention to different kinds of advertising.