Terry Pratchett - Science of Discworld

We can use a sample space approach to get a ball-park estimate of how amazing the Formula One coincidence was. Top drivers all lap at very nearly the same speed, so the three fastest times can eas­ily fall inside the same tenth-of-a-second period. At intervals of a thousandth of a second, there are one hundred possible lap times for each to 'choose' from: this list determines the sample space, The probability of the coincidence turns out to be one chance in ten thousand. Unlikely enough to be striking, but not so unlikely that we ought to feel amazed.

Estimates like this help to explain astounding coincidences reported in newspapers, such as a bridge player getting a 'perfect hand', all thirteen cards in one suit. The number of games of bridge played every week worldwide is huge, so huge that every few weeks the actual events explore the entire sample space. So occa­sionally a perfect hand actually does turn up, with the frequency that its small but non-zero probability predicts. The probability of all four players getting a perfect hand at the same time, though, is so micoscopic that even if every planet in the galaxy had a billion inhabitants, all playing bridge every day for a billion years, you wouldn't expect it to happen.

Nevertheless, every so often the newspapers report a four-way perfect hand. The sensible conclusion is not that a miracle hap­pened, but that something changed the odds. Possibly the players got close to a four-way perfect hand, and the tale grew in the telling, so that when the journalist arrived with a photographer, another kind of narrative imperative ensured that their story fitted what the jour­nalist had been told. Possibly they deliberately cheated to get their names in the papers. Scientists, especially, tend to underestimate the propensity of people to lie. More than one scientist has been fooled into accepting apparent evidence of extrasensory perception or other 'supernatural' events, which can actually be traced to delib­erate trickery.

Many other apparent coincidences, on close investigation, slither into a grey area in which trickery is strongly suspected, but may never be proved, either because sufficient evidence is unob­tainable, or because it's not worth the trouble. Another way to be fooled about a coincidence is to be unaware of hidden constraints that limit the sample space. That 'perfect hand' could perhaps be explained by the way bridge players often shuffle cards for the next deal, which can be summed up as: poorly. If a pack of cards is arranged so that the top four cards consist of one from each suit, and thereafter every fourth card is in the same suit, then you can cut (but not shuffle, admittedly) the pack as many times as you like, and it will deal out a four-way perfect hand. At the end of a game, the cards lie on the table in a fairly ordered manner, not a random one, so it's not so surprising if they possess a degree of structure after they've been picked up.

So even with a mathematically tidy example like bridge, the choice of the 'right' sample space is not entirely straightforward. The actual sample space is 'packs of cards of the kind that bridge players habitually assemble after concluding a game', not 'all possi­ble packs of cards'. That changes the odds.

Unfortunately, statisticians tend to work with the 'obvious' sam­ple space. For that question about Israeli fighter pilots, for instance, they would naturally take the sample space to be all children of Israeli fighter pilots. But that might well be the wrong choice, as the next tale illustrates.

According to Scandinavian folklore, King Olaf of Norway was in dispute with the King of Sweden about ownership of an island, and they agreed to throw dice for it: two dice, highest total wins. The Swedish king threw a double-six. 'You may as well give up now,' he declared in triumph. Undeterred, Olaf threw the dice ... One turned up six ... the other split in half, one face showing a six and the other a one. 'Thirteen, I win,' said Olaf.'

Something similar occurs in The Colour ofMagic, where several gods are playing dice to decide certain events on the Discworld:

The Lady nodded slightly. She picked up the dice-cup and held it steady as a rock, yet all the Gods could hear the three cubes rattling about inside. And then she sent them bouncing across the table.

A six. A three. A five.

Something was happening to the five, however. Battered by the chance collision of several billion molecules, the die flipped onto a point, spun gently and came down a seven.

Blind lo picked up the cube and counted the sides.

'Come on ,' he said wearily. 'Play fair.'

Nature's sample space is often bigger than a conventional statisti­cian would expect. Sample spaces are a human way to model reality: they do not capture all of it. And when it comes to estimating sig­nificance, a different choice of sample space can completely change our estimates of probabilities. The reason for this is an extremely important factor, 'selective reporting', which is a type of narra-tivium in action. This factor tends to be ignored in most conventional statistics. That perfect hand at bridge, for instance, is far more likely to make it to the local or even national press than an imperfect one. How often do you see the headline BRIDGE PLAYER GETS ENTIRELY ORDINARY HAND, for instance? The human brain is an irrepressible pattern-seeking device, and it seizes on certain events that it considers significant, whether or not they really are. In so doing, it ignores all the 'neighbouring' events that would help it judge how likely or unlikely the perceived coincidence actually is.

Selective reporting affects the significance of those Formula One times. If it hadn't been them, maybe the tennis scores in the US Open would have contained some unusual pattern, or the football results, or the golf ... Any one of those would have been reported, too, but none of the failed coincidences, the ones that didn't hap­pen, would have hit the headlines. FORMULA ONE DRIVERS RECORD DIFFERENT LAP TIMES ... If we include just ten major sporting events in our list of would-be's that weren't, that one in ten thousand chance comes down to only one in a thousand.

Having understood this, let's go back to the Israeli fighter pilots. Conventional statistics would set up the obvious sample space, assign probabilities to boy and girl children, and calculate the chance of getting 84% girls in a purely random trial. If this were less than one in a hundred, say, then the data would be declared 'significant at the 99% level'. But this analysis ignores selective reporting. Why did we look at the sexes of Israeli fighter pilots' chil­dren in the first place? Because our attention had already been drawn to a clump. If instead the clump had been the heights of the children of Israeli aircraft manufacturers, or the musical abilities of the wives of Israeli air traffic controllers, then our clump-seeking brains would again have drawn the fact to our attention. So our computation of the significance level tacitly excludes many other factors that didn 't clump, making it fallacious.

The human brain filters vast quantities of data, seeking things that appear unusual, and only then does it send out a conscious sig­nal: Wow! Look at that! The wider we cast our pattern-seeking net, the more likely it is to catch a clump. For this reason, it's illegitimate to include the data that brought the clump to our attention as part of the evidence that the same clump is unusual. It would be like sorting through a pack of cards until you found the ace of spades, putting it on the table, and then claiming miraculous powers that unerringly accomplish a feat whose probability is one in 52.

Exactly this error was made in early experiments on extra-sen­sory perception. Thousands of subjects were asked to guess cards from a special pack of five symbols. Anyone whose success rate was above average was invited back, while the others were sent home. After this had gone on for several weeks, the survivors all had an amazing record of success! Then these 'good guessers' were tested some more. Strangely, as time went on, their success rate slowly dropped back towards the average, as if their powers were 'running down'. Actually, that effect wasn't strange at all. It happened because the initial high scores were included in the running total. If they had been omitted, then the scoring rate would have dropped, immediately, to near average.