Iain K. Crombie - The Pocket Guide to Critical Appraisal

The critical appraisal of a paper is conducted in three stages: identify the research design, select the appropriate appraisal tool for that study, and apply the tool to assess the bias and the value of the research. This section introduces these stages, with subsequent chapters explaining in detail how to carry them out.

Research studies can ask different types of questions. Some may be concerned with the effectiveness of treatments; others may investigate the likely prognosis of an illness. The research design should be matched to the research question. To evaluate treatment effectiveness a randomised controlled trial would be used, whereas prognosis would be investigated by a cohort study. The other designs are surveys, case–control studies, cohort studies that test interventions, and systematic reviews. Chapter 3of this book provides guidance on identifying the correct design.

Critical appraisal tools have been developed to enable a forensic evaluation of research studies. They provide lists of questions that probe all aspects of the methods used and the results obtained. Judgements can then be made about the bias and value of the research. This book presents appraisal tools for the six most common research designs in medical research ( Chapters 6– 11).

A key feature of the questions on bias is that they distinguish between critical and important sources of bias. For each research design there are a few crucial steps which, if conducted poorly, are very likely to lead to misleading findings. Flaws in the other steps are less likely to cause serious bias. Only when several of these flaws occur are serious concerns raised.

Value, the usefulness or worth of a finding to individuals and to society, is more difficult to assess. Assessing the potential for improving health is usually straightforward, but often some pieces of information, such as the importance of the findings to patients, are not available. Decisions about value can use information from several sources and require judgement to balance many complex issues.

The critical appraisal questions can only be answered by identifying key pieces of information from relevant sections of a paper. Chapter 2provides a simple method for extracting the key pieces of information from a research paper. Careful reflection on what the questions reveal enables an assessment of the quality of the research study.

The appraisal questions presented in this book are structured to lead to one of three answers: yes, no, or not enough information to decide. Selecting the appropriate option is not a simple tick box process. Critical appraisal involves pausing to think about the implications of each limitation, and whether it could seriously affect the interpretation of the study's findings. Sometimes there will be insufficient information to answer a question. As decisions in medicine affect patients, it is usually better to be cautious in drawing conclusions.

In summary, critical appraisal assesses the bias and the value of research studies. By identifying the research design, the appropriate appraisal tool is selected. It asks probing questions about the design of the study and the way it was conducted, analysed, and interpreted. Chapter 2describes a simple method for extracting the information from published papers to facilitate critical appraisal.

The most important piece of advice when appraising a scientific paper is: do not read the paper. Scientific papers are complicated, information dense, and full of technical jargon – they cannot be read like a newspaper article. They are written for experts and prioritise technical correctness over ease of understanding. The research design is often complex, and the statistical analysis sophisticated. Papers are written with the assumption that the reader is familiar with the scientific terms and the methodologies used. Reading every word from the Introduction to the Discussion may lead to confusion.

A better approach is to interrogate the paper, asking questions to make it reveal the information about study design and conduct. Unlike reading, which is a passive process, critical appraisal searches out the key pieces of information about a study. Interrogation puts the reader in charge, directing questions at different sections of the paper, identifying and checking the information in the paper. This approach provides a quicker appraisal of a paper because it focuses on the important features of research studies. It probes not just what was done, but how well it was done. Sometimes the authors of papers try to gloss over weaknesses in their study. Interrogation will identify flaws that reading may miss. It provides all the information needed to evaluate the risk of bias and value of the results.

The initial interrogation involves delving into some sections of the paper to get a feel for what the study is about. The aim is to start constructing a mental map of the study, not to engage in a serious appraisal. The main benefit of the map is that it allows you to slot subsequent pieces of information into a mental framework. This simplifies the task of understanding the paper and helps you organise the information it contains. The process uses four questions.

The interrogation begins with the title. Good titles should indicate the patient group being investigated, the research design used, and why the study is being done. Sometimes titles are written to be catchy rather than informative, but even then, they should provide one or two facts about the study. The title provides the first indication of what the study is about.

The Abstract usually gives key information about the four main sections of the paper: the Introduction, Methods, Results, and Discussion. Ideally it will clarify the study aim, the research design that was used, the main findings and their implications. Together with the title, the Abstract should enable you to construct a provisional map of the paper. However, this is not guaranteed. Information in the Abstracts may be presented to impress rather than to be informative. The provisional map should be tested by comparing what it says with the information presented in the Introduction and the Methods sections.

The aims of the study are usually presented towards the end of the Introduction. They may be phrased as hypotheses to be tested or as questions to be answered. The absence of a clear statement of aims could mean that the authors had no real idea of what they were trying to find out. Research studies with unclearly stated aims are often of poor quality.

The Methods section should explain how all aspects of the study were carried out. The descriptions in this section are often brief and difficult to follow. This section should clarify how participants were recruited, the data that were collected, and the statistical techniques that were used. Often, the research design, which should explain how the study aims will be achieved, is not clearly stated. To assist with this limitation, Chapter 3provides guidance on identifying the research design. It outlines the main features and the key terms indicative of the six most common research designs.