Iain K. Crombie - Evidence in Medicine

This chapter has explored the many sources of bias that commonly afflict randomised controlled trials. The randomised controlled trial is held to be the gold standard for evidence on treatment effectiveness, but that gold is more than a little tarnished. One commentator concluded that randomised controlled trials are: ‘often flawed, mostly useless, clearly indispensable’ [106].

The two key questions for clinical trials are: how frequently do these flaws occur; and how big an effect do they exert on estimated effect sizes? The frequency of flaws varies across individual review studies and by the type of deficiency, so there is not a specific estimate of how often bias occurs. Instead we can put the frequencies on a scale from very rare to very common. As most of the estimates from review articles are in the fairly common or very common area, flaws are a serious problem.

The magnitude of bias from randomisation and allocation concealment is generally modest, amounting to a 10%–15% increase in the estimated treatment effect. However these estimates are averages based on large numbers of trials that cover many different types of treatment. It is likely that the impact of bias is much larger for some trials than others, although we do not know which.

Some of the weaknesses in clinical trials may simply be due to lack of knowledge or experience. This could explain deficiencies in the handling loss to follow‐up or the lack of blinding of outcome assessment. Tampering with the randomisation sequence, and the replacement of primary outcomes, suggests a less innocent explanation. The deficiencies described in this chapter, which may result from inadvertent mistakes or deliberate actions, pose a serious threat to the integrity of medical evidence. The next chapter describes weaknesses in study design and conduct that lead to wasted and unhelpful trials.

1 1. Riechelmann, R.P., Peron, J., Seruga, B. et al. (2018). Meta‐research on oncology trials: a toolkit for researchers with limited resources. Oncologist 23: 1467–1473.

2 2. Page, M.J., Higgins, J.P., Clayton, G. et al. (2016). Empirical evidence of study design biases in randomized trials: systematic review of meta‐epidemiological studies. PLoS One https://doi.org/10.1371/journal.pone.0159267.

3 3. Adie, S., Harris, I.A., Naylor, J.M. et al. (2017). The quality of surgical versus non‐surgical randomized controlled trials. Contemp. Clin. Trials Commun. 5: 63–66.

4 4. Savovic, J., Jones, H., Altman, D. et al. (2012). Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta‐epidemiological studies. Health Technol. Assess. 16: 1–82.

5 5. Dechartres, A., Trinquart, L., Atal, I. et al. (2017). Evolution of poor reporting and inadequate methods over time in 20 920 randomised controlled trials included in Cochrane reviews: research on research study. BMJ https://doi.org/10.1136/bmj.j2490.

6 6. Wuytack, F., Regan, M., Biesty, L. et al. (2019). Risk of bias assessment of sequence generation: a study of 100 systematic reviews of trials. Syst. Rev. https://doi.org/10.1186/s13643‐018‐0924‐1.

7 7. Savovic, J., Turner, R.M., Mawdsley, D. et al. (2018). Association between risk‐of‐bias assessments and results of randomized trials in Cochrane reviews: the ROBES meta‐epidemiologic study. Am. J. Epidemiol. 187: 1113–1122.

8 8. Zhai, X., Cui, J., Wang, Y. et al. (2017). Quality of reporting randomized controlled trials in five leading neurology journals in 2008 and 2013 using the modified “risk of bias” tool. World Neurosurg. 99: 687–694.

9 9. Rikos, D., Dardiotis, E., Tsivgoulis, G. et al. (2016). Reporting quality of randomized‐controlled trials in multiple sclerosis from 2000 to 2015, based on CONSORT statement. Mult. Scler. Relat. Disord. 9: 135–139.

10 10. Saltaji, H., Armijo‐Olivo, S., Cummings, G.G. et al. (2018). Impact of selection bias on treatment effect size estimates in randomized trials of Oral health interventions: a meta‐epidemiological study. J. Dent. Res. 97: 5–13.

11 11. Schulz, K.F. and Grimes, D.A. (2002). Unequal group sizes in randomised trials: guarding against guessing. Lancet 359: 966–970.

12 12. Clark, L., Fairhurst, C., Hewitt, C.E. et al. (2014). A methodological review of recent meta‐analyses has found significant heterogeneity in age between randomized groups. J. Clin. Epidemiol. 67: 1016–1024.

13 13. Clark, L., Fairhurst, C., Cook, E. et al. (2015). Important outcome predictors showed greater baseline heterogeneity than age in two systematic reviews. J. Clin. Epidemiol. 68: 175–181.

14 14. Schulz, KF. (1995). Subverting randomization in controlled trials. JAMA 274: 1456–1458.

15 15. Paludan‐Muller, A., Laursen, D.R.T., and Hrobjartsson, A. (2016). Mechanisms and direction of allocation bias in randomised clinical trials. BMC Med. Res. Methodol. https://doi.org/10.1186/s12874‐016‐0235‐y.

16 16. Clark, L., Fairhurst, C., and Torgerson, D.J. (2016). Allocation concealment in randomised controlled trials: are we getting better? BMJ https://doi.org/10.1136/bmj.i5663.

17 17. Schulz, K.F., Chalmers, I., Hayes, R.J. et al. (1995). Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273: 408–412.

18 18. Dechartres, A., Trinquart, L., Faber, T. et al. (2016). Empirical evaluation of which trial characteristics are associated with treatment effect estimates. J. Clin. Epidemiol. 77: 24–37.

19 19. Pansieri, C., Pandolfini, C., and Bonati, M. (2015). The evolution in registration of clinical trials: a chronicle of the historical calls and current initiatives promoting transparency. Eur. J. Clin. Pharmacol. 71: 1159–1164.

20 20. Zarin, D.A., Tse, T., Williams, R.J. et al. (2017). Update on trial registration 11 years after the ICMJE policy was established. N. Engl. J. Med. 376: 383–391.

21 21. Howard, B., Scott, J.T., Blubaugh, M. et al. (2017). Systematic review: outcome reporting bias is a problem in high impact factor neurology journals. PLoS One https://doi.org/10.1371/journal.pone.0180986.

22 22. Wayant, C., Scheckel, C., Hicks, C. et al. (2017). Evidence of selective reporting bias in hematology journals: a systematic review. PLoS One https://doi.org/10.1371/journal.pone.0178379.

23 23. Hannink, G., Gooszen, H.G., and Rovers, M.M. (2013). Comparison of registered and published primary outcomes in randomized clinical trials of surgical interventions. Ann. Surg. 257: 818–823.

24 24. Raghav, K.P., Mahajan, S., Yao, J.C. et al. (2015). From protocols to publications: a study in selective reporting of outcomes in randomized trials in oncology. J. Clin. Oncol. 33: 3583–3590.

25 25. Dwan, K., Gamble, C., Williamson, P.R. et al. (2013). Systematic review of the empirical evidence of study publication bias and outcome reporting bias – an updated review. PLoS One https://doi.org/10.1371/journal.pone.0066844.

26 26. Li, G., Abbade, L.P.F., Nwosu, I. et al. (2018). A systematic review of comparisons between protocols or registrations and full reports in primary biomedical research. BMC Med. Res. Methodol. https://doi.org/10.1186/s12874‐017‐0465‐7.

27 27. Chen, T., Li, C., Qin, R. et al. (2019). Comparison of clinical trial changes in primary outcome and reported intervention effect size between trial registration and publication. JAMA Netw. Open https://doi.org/10.1001/jamanetworkopen.2019.7242.

28 28. Smyth, R.M., Kirkham, J.J., Jacoby, A. et al. (2011). Frequency and reasons for outcome reporting bias in clinical trials: interviews with trialists. BMJ https://doi.org/10.1136/bmj.c7153.