Iain K. Crombie - The Pocket Guide to Critical Appraisal

While the publisher and authors have used their best efforts in preparing this work, they make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives, written sales materials or promotional statements for this work. The fact that an organization, website, or product is referred to in this work as a citation and/or potential source of further information does not mean that the publisher and authors endorse the information or services the organization, website, or product may provide or recommendations it may make. This work is sold with the understanding that the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for your situation. You should consult with a specialist where appropriate. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

Library of Congress Cataloging‐in‐Publication Data Applied for

Paperback ISBN: 9781119835240

Cover Design: Wiley

Cover Image: © Govindanmarudhai/Getty Images

I am grateful to the University of Dundee for providing the facilities to research and write the second edition of this book. The success of the first edition was due in no small part to my colleagues for their support and encouragement. As the style and much of the content of this new edition follows the original, I am pleased to be able to thank them again for their help. Special thanks are due to Geraldine Fardon whose constructive comments highlighted imprecise, vague, and confusing text. I take full responsibility for any remaining flaws.

This book was written to meet the needs of health professionals as medicine moves to be evidence‐based. The initial idea arose during discussion with younger colleagues and students on difficulties of interpreting the medical literature. It quickly became apparent that their needs would be best met by a short book detailing criteria for critical appraisal.

The book is organised in two parts. The first five chapters provide an introduction to critical appraisal of quantitative research, indicating how papers can be read and how the results can be interpreted. Experienced researchers could easily omit these chapters. The final six chapters provide annotated checklists for critical appraisal. The first of these contains the general questions which can be asked of any study using a quantitative methodology. The succeeding five chapters review in turn the questions which are specific for each quantitative method. For convenience each of these five chapters concludes with a combined list of general and specific questions.

The book has been written to be simple and quick to use. Technical terms are avoided where possible, and the assessment criteria are explained but not justified. A larger and less accessible text would have been needed to give a proper rationale for each the checklists. To keep this a pocket guide, it was also decided to omit evaluations of other topics such as qualitative methods, health economics, clinical audit, decision analysis, and screening tests. An argument could be made for the inclusion of each, but to include them all would nearly double the size of the book. I hope the checklists prove useful.

Several developments have highlighted the need for a second edition of this book. Recent work, particularly from the new discipline of meta‐research, has provided important evidence for critical appraisal. By examining and comparing large numbers of studies, this research has clarified the nature of and impact of many sources of bias. A separate development is the recognition that critical appraisal should distinguish between the risk of bias and the value of the findings to individuals, health care systems, and the wider society. Critical appraisal checklists should provide separate sets of questions for the two issues. Finally, many years of teaching critical appraisal skills have provided the author with an understanding of the difficulties that students and health professionals encounter when evaluating papers. These developments have led to improvements in the guidance given in this book.

This new edition follows the format and the spirit of the first: it is written to be easy to use and, where possible, technical terms are avoided. However, substantial modifications have been made. A new Introduction clarifies the aims of critical appraisal. The second chapter, entitled ‘Do Not Read the Paper’, is also new; it provides a simple method for eliciting the information needed for critical appraisal from published studies. This is supported by Chapter 5, ‘The In‐Depth Interrogation’, which is a much‐amended version of the original ‘Standard Appraisal Questions’. In addition, Chapter 4on ‘Interpreting the Results’ has been updated in line with current views on the nature of statistical significance and the interpretation of confidence intervals.

All of the chapters that provide checklists for critical appraisal of specific research designs have been revised and updated, particularly Chapter 9, ‘Appraising Randomised Controlled Trials’, and Chapter 11, ‘Appraising Systematic Reviews’. A new chapter for the critical appraisal of cohort studies that evaluate interventions has been added. This research design is now being frequently used in medical research (the separate chapter on conventional cohort studies has been retained). All the checklists distinguish between critical and important sources of bias and they evaluate risk of bias separately from value.

Two new chapters at the end of the book provide methods for synthesising the findings of critical appraisal. One provides a method for summarising risk of bias. The second explores the important concept of certainty of evidence and describes how it is assessed. A further new chapter reviews the wide range of factors that determine the value of research findings to society as a whole. This second edition provides a much‐needed update to the guidance on critical appraisal.

Critical appraisal is the process of assessing the strengths and weaknesses of published studies. It involves a structured and rigorous evaluation of all the stages of the research, from design to analysis and interpretation. All studies have imperfections, so the question is not if there are flaws, but whether they are likely to be important. Critical appraisal is a method of systematically examining research studies to determine the worth of their findings.

Critical appraisal assesses two issues: bias in study design or conduct, and the value that the findings have for clinical practice. Bias occurs when systematic errors distort the results of a study. For example, a clinical trial should provide a good estimate of the benefit of a treatment, but weaknesses in design or conduct could produce a misleading estimate. There are many types of bias and these are explored through a series of questions that examine the main deficiencies in research.

The value of findings is the extent to which they will provide benefits for patients or the general population. Many factors contribute to the value of research findings. For a clinical trial, this would include the size of the treatment benefit, the importance of the outcome measure to patients, and whether the patients to be treated in a clinical setting are similar to the participants in the study. As with bias, there are sets of questions that assess value.

Читать дальше