Paul J. Mitchell - Experimental Design and Statistical Analysis for Pharmacology and the Biomedical Sciences

The principle problem here is ignorance that rigorous statistical analysis is a component of good experimental design. Consequently, the statistical methodology to be employed in research must be decided before the experiments are performed. In my experience, this is due to historically very poor teaching of statistics in pharmacology across the sector, such that those now with the responsibility of teaching pharmacology to current undergraduates or newly qualified graduates (whether they be in academia or the pharmaceutical industry) are themselves at a disadvantage and too naive to understand the importance of rigorous statistical analysis. Consequently, they are unable to provide high‐quality supervision to enable less experienced individuals to develop and hone their experimental technique.

I was once stopped in the corridor by a fellow post‐doc (and close friend) who described a series of experiments involving cell culture in different mediums which they were unsure how to analyse. Essentially, the post‐doc had a single flask of a particular CHO cell line and was trying to determine which of three mediums promoted the best cell growth. Three further flasks were prepared each one containing a different medium, and a sample of the cell line was decanted into each of the three test flasks. Sometime later, three samples were taken from each flask (so nine samples in total) and the number of cells per unit volume determined. The question was; ‘how do I analyse the data? Do I do a number of t ‐tests (are they paired)? Do I do ANOVA? And if so, which post hoc test ( don't worry I'll explain all these terms later in the book )’. I looked at the data, checked I had the right information about the design of this simple experiment and said ‘Sorry, you don't have enough data for statistical analysis – you only have an n of one in each case’. The post‐doc stared at me quizzically and said, ‘Don't be daft, I have n of 3 for each medium!’. ‘Er…, no!’, I replied, ‘You estimated the cell numbers in triplicate, but that only gives you n = 1 in each case, all you've done is obtain an estimate of precision and hopefully accuracy of your estimates, but that doesn't change the fact that you've only got n = 1 for each flask’. ‘No, no, no!’ the post‐doc strenuously exclaimed, ‘I have n = 3 in each case, three samples from each flask for the different mediums!’. ‘Er, no!’, I replied (at the risk of repeating myself), ‘If you wanted to do this properly then you should have prepared three flasks for each medium (so nine flasks in total), and decanted the same volume of CHO cells into each flask. Sometime later, you should then have taken 3 samples from each flask (so 27 samples) and estimated the cell number in each case. You would then calculate the average for each flask so that you get an improved accurate measure of the cell concentration in each flask (thanks to the measures in triplicate). This will then give you three measures for each medium which you can analyse by one‐way ANOVA followed by a Tukey All Means post hoc test ( don't worry about these terms; all will become clear later in the book. I just included them to impress you, whet your appetite for what is to come and to try and convince you I know what I'm talking about! ). The post‐doc looked at me aghast! ‘I don't have time for that!’, came the reply, ‘I have a group meeting with my Prof. this afternoon and I need to present this data so we can discuss which medium to use in our future studies – our latest grant proposal depends on demonstrating that one of these mediums is significantly different from the others, so I need to subject these data to statistical analysis!’. I looked at the summary bar chart the post‐doc had prepared from the data and it was clear from the eye‐ball test ( this is probably one of the best tests to use to appreciate data and is very simple to perform – I'll reveal how later in the book! ) that one of the mediums showed clear advantages in terms of cell growth than the others. ‘Just look at your data’, I said, ‘Medium X is clearly better than Y or Z , that's all you have to say!’. (This statement refers to one of the other very common statistical test; the IBO test. Er…, IBO is defined as ‘It's Bloody Obvious’). ‘That's no good’, replied the post doc ‘Prof likes to see stars ( so thump him, I thought but didn't say out loud ), and the more stars the better and happier he'll be’. ( So, thump him harder? At this point, I'm left with the thought, what is the difference between a five years old starting Primary School and a 50 years old Prof leading a prestigious research group? Answer = absolutely nothing, they both like stars, preferably gold! ] ‘Not with these data’, I said resignedly, ‘you've only got n = 1 for each medium and you can't perform rigorous statistical analysis with such a paucity of data.’ ‘Never mind’ said the post doc ‘I'll do it my way. Thanks.’ and disappeared down the corridor repeatedly muttering ‘1‐way ANOVA and some bloody post hoc test’. My words were left hanging in the ether!

The problem here is ignorance about n numbers leading to poor experimental design, a lack of understanding of the term ‘triplicate’ and the consequences that this has for statistical analysis. This is a very simple error that is commonly seen.

Publication of research in scientific journals depend on the peer review system (where experienced researchers act as referees on papers submitted to journals for publication). A friend of mine excitedly showed me a paper he had just been requested to review from a very reputable journal. He was thrilled on two counts; first, the fact that he had been asked to act as referee by this journal was a form of tacit recognition of his own expertise. Second, the paper was written by a group he had long admired and indeed, I think he hoped he could join this group in the future to further his own career. The following day we met, and he looked very downcast. It transpired that he had spent a couple of hours looking through the manuscript the previous evening and it soon became apparent that the statistical analysis used by the authors was not appropriate according to the experimental design and the type of data generated by the studies described. It was clear that he was going to have to reject the paper in its current state and advise that publication would only be acceptable if the statistical analysis was completely revised. He duly submitted his report to the journal. I didn't hear anything more for a month or so, but eventually he showed me a letter he had subsequently received from the journal. The letter thanked him for his time and effort in reviewing the manuscript, the tactful way he had written his review and the conscientious and constructive way he had dealt with the statistical shortcomings of the paper. Included also were the comments made by other referees (they were all in agreement in rejecting the manuscript and for the same reasons) and the subsequent response from the authors. To cut a long story short, the authors rejected the comment made by the editor and referees and, in particular, were scathing about the comments regarding their statistical analysis. Their final comment was (and I paraphrase here), ‘This is the way we've always analysed our data. We see no reason to change now and so we'll continue to do it our way!’. The paper was rightly rejected by the journal. However, six months later (or so) my friend barged into my office and threw a paper on my desk. ‘Look at this!’, he exclaimed (getting increasingly louder as he muttered words which I can't repeat here but which seriously questioned the marital status of the parents of those unfortunate individuals related in some way to the document now lying in front of me). The paper was the manuscript he (and others) had rejected earlier but published almost word for word in a different journal! So, while it may be difficult to teach a dog new tricks (see above), success probably depends on the dog willing to learn something afresh – it's just unfortunate that some dogs are too arrogant and set in their ways to consider and adopt new methods or to accept that perhaps what they did in the past was not the best approach.