Iain K. Crombie - Evidence in Medicine

Editorial Office 9600 Garsington Road, Oxford, OX4 2DQ, UK

For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com.

Wiley also publishes its books in a variety of electronic formats and by print‐on‐demand. Some content that appears in standard print versions of this book may not be available in other formats.

Limit of Liability/Disclaimer of Warranty The contents of this work are intended to further general scientific research, understanding and discussion only and are not intended and should not be relied upon as recommending or promoting scientific method, diagnosis or treatment by physicians for any particular patient. In view of ongoing research, equipment modifications, changes in governmental regulations and the constant flow of information relating to the use of medicines, equipment and devices, the reader is urged to review and evaluate the information provided in the package insert or instructions for each medicine, equipment or device for, among other things, any changes in the instructions or indication of usage and for added warnings and precautions. While the publisher and authors have used their best efforts in preparing this work, they make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives, written sales materials or promotional statements for this work. The fact that an organisation, website or product is referred to in this work as a citation and/or potential source of further information does not mean that the publisher and authors endorse the information or services the organisation, website or product may provide or recommendations it may make. This work is sold with the understanding that the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for your situation. You should consult with a specialist where appropriate. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential or other damages.

Library of Congress Cataloging‐in‐Publication Data

Names: Crombie, I. K., author.

Title: Evidence in medicine : the common flaws, why they occur and how to prevent them / Iain K. Crombie.

Description: First edition. | Hoboken, NJ : John Wiley & Sons, Inc., 2021. | Includes bibliographical references and index.

Identifiers: LCCN 2020051750 (print) | LCCN 2020051751 (ebook) | ISBN 9781119794141 (paperback) | ISBN 9781119794189 (adobe pdf) | ISBN 9781119794196 (epub)

Subjects: MESH: Research Design–standards | Randomized Controlled Trials as Topic | Systematic Reviews as Topic | Evidence‐Based Practice–standards | Treatment Outcome

Classification: LCC R852 (print) | LCC R852 (ebook) | NLM W 20.5 | DDC 610.72–dc23

LC record available at https://lccn.loc.gov/2020051750LC ebook record available at https://lccn.loc.gov/2020051751

Cover Design: Wiley

Cover Image: © Govindanmarudhai/DigitalVision Vectors/Getty Images

Evidence is central to the practice of medicine. The amount of published medical evidence is immense, but there are widespread concerns about the quality of many studies. Deficiencies in the conduct of research studies often result in misleading estimates of the benefits of treatments, so that ineffective treatments may be used in clinical practice. In addition, many studies are wasted because they have been so poorly designed and conducted.

This book explores the nature of the deficiencies and flaws in the evidence about the effectiveness of treatments. It is the result of a career‐long interest in medical evidence. A recent career change provided an opportunity to read the extensive literature on research quality that has emerged in recent years. As befits a book on evidence, references are cited to support the statements made. Reviews and commentaries are used where available, although many landmark studies are also referenced. The approach taken is to cite sufficient papers to support a point, rather than give a comprehensive a review of it. The number of references cited reflect the wealth of evidence on the deficiencies in medical research.

I am grateful to the University of Dundee for providing the facilities to research and write this book, and to my colleagues for their support and encouragement. Special thanks are due to my long‐time colleague and friend Linda Irvine: she made several gently delivered, trenchant criticisms of early drafts and picked out factual, logical and grammatical failings in later drafts. The irony of writing a book about evidence is that it may contain errors of fact or interpretation. Equally there may be errors of omission. I take full responsibility for all flaws.

The advent of the randomised controlled trial (RCT) provided a method to generate reliable evidence on the effectiveness of treatments. It enables a fair comparison of groups [1], and should identify which treatments are beneficial and which are of little or no value. Since the early randomised controlled trials (RCTs) of the mid‐twentieth century, the design of trials has been progressively refined, making them the bedrock on which modern medicine is built. The findings of RCTs are used by regulatory authorities across the world to licence effective treatments. Other methods of testing treatments have been proposed, but none matches the ability of high quality randomised trials to provide good estimates of treatment effectiveness [2]. The RCT deserves its status as the gold standard for assessing the effectiveness of treatment.

The RCT now occupies a leading place in medical research, with tens of thousands of trials being published annually [3, 4]. These studies should provide high quality evidence across all fields of medicine, but that promise has not been fulfilled. At issue is the quality of the evidence. Concerns are growing ‘about the reliability and validity of the underlying research that supports regulatory and clinical decision‐making’ [5], with authors describing ‘the pervasiveness of poor quality clinical evidence’ [6] and concluding that ‘much of the published medical research is apparently flawed, cannot be replicated and/or has limited or no utility’ [7]. In a particularly trenchant comment, Ian Roberts and colleagues concluded that ‘the knowledge system underpinning healthcare is not fit for purpose and must change’ [8]. This book evaluates that proposition.

To investigate the knowledge system for healthcare, this book poses five questions: 1) what are the problems; 2) how common are they; 3) to what extent do they bias evidence; 4) why do they occur, and 5) how can they be prevented? Medical research is a vast enterprise, so the book focuses on the two most important research methods: the randomised controlled trial and the systematic review. Answers to the first three questions will provide a broad assessment of the quality of evidence ( Chapters 2– 5). The answer to the fourth question, on the causes of poor quality, highlights issues around misconduct and how the structures and incentives in the research environment influence the quality of evidence ( Chapters 6and 7). The final chapter presents an approach for developing a comprehensive strategy to the quality problem. This is supported by an Appendix, which lists the initiatives that have been proposed to improve research evidence. First, to introduce the nature of evidence in medicine, Chapter 1provides a brief review of the rationale for treatments from ancient times to the present day.