Ben Goldacre - Bad Science

It’s worth going through these misrepresentations of evidence, mainly because they are fascinating illustrations of how people can get things wrong, but also because the aim of this book is that you should be future-proofed against new variants of bullshit. There are also two things we should be very clear on. Firstly, I’m picking out individual examples as props, but these are characteristic of the genre; I could have used many more. Nobody is being bullied, and none of them should be imagined to stand out from the nutritionist crowd, although I’m sure some of the people covered here won’t be able to understand how they’ve done anything wrong.

Secondly, I am not deriding simple, sensible, healthy eating advice. A straightforwardly healthy diet, along with many other aspects of lifestyle (many of which are probably more important, not that you’d know it from reading the papers) is very important. But the media nutritionists speak beyond the evidence: often it is about selling pills; sometimes it is about selling dietary fads, or new diagnoses, or fostering dependence; but it is always driven by their desire to create a market for themselves, in which they are the expert, whereas you are merely bamboozled and ignorant.

Prepare to switch roles.

The four key errors

Does the data exist?

This is perhaps the simplest canard of all, and it happens with surprising frequency, in some rather authoritative venues. Here is Michael van Straten on BBC Newsnight , talking ‘fact’. If you prefer not to take it on faith that his delivery is earnest, definitive, and perhaps even slightly patrician, you can watch the clip online.

‘When Michael van Straten started writing about the magical medicinal powers of fruit juices, he was considered a crank,’ Newsnight begins. ‘But now he finds he’s at the forefront of fashion.’ (In a world where journalists seem to struggle with science, we should note that Newsnight has ‘crank’ at one end of the axis, and ‘fashion’ at the other. But that chapter comes later.) Van Straten hands the reporter a glass of juice. ‘Two years added to your life expectancy in that!’ he chuckles – then a moment of seriousness: ‘Well, six months, being honest about it.’ A correction. ‘A recent study just published last week in America showed that eating pomegranates, pomegranate juice, can actually protect you against ageing, against wrinkles,’ he says.

Hearing this on Newsnight , the viewer might naturally conclude that a study has recently been published in America showing that pomegranates can protect against ageing. But if you go to Medline, the standard search tool for finding medical academic papers, no such study exists, or at least not that I can find. Perhaps there’s some kind of leaflet from the pomegranate industry doing the rounds. He goes on: ‘There’s a whole group of plastic surgeons in the States who’ve done a study giving some women pomegranates to eat, and juice to drink, after plastic surgery and before plastic surgery: and they heal in half the time, with half the complications, and no visible wrinkles!’ Again, it’s a very specific claim – a human trial on pomegranates and surgery – and again, there is nothing in the studies database.

So could you fairly characterise this Newsnight performance as ‘lying’? Absolutely not. In defence of almost all nutritionists, I would argue that they lack the academic experience, the ill-will, and perhaps even the intellectual horsepower necessary to be fairly derided as liars. The philosopher Professor Harry Frankfurt of Princeton University discusses this issue at length in his classic 1986 essay ‘On Bullshit’. Under his model, ‘bullshit’ is a form of falsehood distinct from lying: the liar knows and cares about the truth, but deliberately sets out to mislead; the truth-speaker knows the truth and is trying to give it to us; the bullshitter, meanwhile, does not care about the truth, and is simply trying to impress us:

It is impossible for someone to lie unless he thinks he knows the truth. Producing bullshit requires no such conviction … When an honest man speaks, he says only what he believes to be true; and for the liar, it is correspondingly indispensable that he considers his statements to be false. For the bullshitter, however, all these bets are off: he is neither on the side of the true nor on the side of the false. His eye is not on the facts at all, as the eyes of the honest man and of the liar are, except insofar as they may be pertinent to his interest in getting away with what he says. He does not care whether the things he says describe reality correctly. He just picks them out, or makes them up, to suit his purpose.

I see van Straten, like many of the subjects in this book, very much in the ‘bullshitting’ camp. Is it unfair for me to pick out this one man? Perhaps. In biology fieldwork, you throw a wired square called a ‘quadrat’ at random out onto the ground, and then examine whatever species fall underneath it. This is the approach I have taken with nutritionists, and until I have a Department of Pseudoscience Studies with an army of PhD students doing quantitative work on who is the worst, we shall never know. Van Straten seems like a nice, friendly guy. But we have to start somewhere.

Observation, or intervention?

Does the cock’s crow cause the sun to rise? No. Does this light switch make the room get brighter? Yes. Things can happen at roughly the same time, but that is weak, circumstantial evidence for causation. Yet it’s exactly this kind of evidence that is used by media nutritionists as confident proof of their claims in our second major canard.

According to the Daily Mirror , Angela Dowden, RNutr, is ‘Britain’s Leading Nutritionist’, a monicker it continues to use even though she has been censured by the Nutrition Society for making a claim in the media with literally no evidence whatsoever. Here is a different and more interesting example from Dowden: a quote from her column in the Mirror , writing about foods offering protection from the sun during a heatwave: ‘An Australian study in 2001 found that olive oil (in combination with fruit, vegetables and pulses) offered measurable protection against skin wrinkling. Eat more olive oil by using it in salad dressings or dip bread in it rather than using butter.’

That’s very specific advice, with a very specific claim, quoting a very specific reference, and with a very authoritative tone. It’s typical of what you get in the papers from media nutritionists. Let’s go to the library and fetch out the paper she refers to (‘Skin wrinkling: can food make a difference?’ Purba MB et al. J Am Coll Nutr . 2001 Feb; 20(1): 71–80). Before we go any further, we should be clear that we are criticising Dowden’s interpretation of this research, and not the research itself, which we assume is a faithful description of the investigative work that was done.

This was an observational study, not an intervention study. It did not give people olive oil for a time and then measure differences in wrinkles: quite the opposite, in fact. It pooled four different groups of people to get a range of diverse lifestyles, including Greeks, Anglo-Celtic Australians and Swedish people, and it found that people who had completely different eating habits – and completely different lives, we might reasonably assume – also had different amounts of wrinkles.

To me this is not a great surprise, and it illustrates a very simple issue in epidemiological research called ‘confounding variables’: these are things which are related both to the outcome you’re measuring (wrinkles) and to the exposure you are measuring (food), but which you haven’t thought of yet. They can confuse an apparently causal relationship, and you have to think of ways to exclude or minimise confounding variables to get to the right answer, or at least be very wary that they are there. In the case of this study, there are almost too many confounding variables to describe.