Individual Participant Data Meta-Analysis

In aggregate data reviews, assessment of risk of bias is usually based on the information available in trial publications and other publicly accessible documents, such as trial registration entries or published protocols, sometimes supplemented by information requested from trial investigators. In an IPD meta‐analysis project, it is common to obtain additional information from protocols, codebooks and forms, or direct from trial investigators, which can increase the clarity of risk of bias assessments compared to those based on trial reports alone. 48,105As discussed in Sections 3.4and 4.2.6, it may be helpful to undertake an initial risk of bias or quality assessment at the planning stages, before considering whether to obtain the IPD. However, the collection of IPD does allow a deeper and more reliable appraisal of data quality and risk of bias than is possible with aggregate data, because there is the opportunity to generate information directly from the IPD. There is also the potential to seek additional or updated trial IPD for inclusion in a meta‐analysis, in order to reduce or remove the potential for bias in particular domains. For example, participants excluded from original trial analyses may be reinstated in the meta‐analysis, or more appropriate statistical methods might be used. Therefore, an overall risk of bias assessment for each trial would be based on whether the design and conduct of the trial, and the quality of its final IPD (after correcting any data errors) are likely to lead to biased results when the IPD are analysed.

While assessing risk of bias in an IPD meta‐analysis involves using many of the same domains and items listed in the RoB 2 tool, 91some items are not relevant, because the IPD circumvents the issue. For example, the availability of IPD avoids reliance on a trial’s original analysis methods and reported results. Table 4.4lists the domains and signalling questions in the RoB 2 tool that are particularly relevant to an IPD meta‐analysis project, and summarises how appraisal of the collected IPD, and other information harnessed through an IPD approach, can inform risk of bias judgements. These individual domains represent a useful starting point for assessing the risk of bias associated with each trial in an IPD meta‐analysis project.

Ultimately, if the trial design, management or conduct are seriously flawed (e.g. failed randomisation, poor follow‐up, or early stopping), this might lead to trials that are wholly or partially unreliable. Depending on the gravity of the issues, a trial may need to be excluded completely from the analyses or be the subject of sensitivity analysis; at the very least, the issues need to be brought to the attention of the reader in the IPD meta‐analysis publication. The IPD project’s central research team should have a plan in place for how they will deal with these sorts of occurrences. However, the objective of data checking is not to police trials or uncover fraud, and in our experience wholesale fabrication of trial data is uncommon. Also, note that this is probably less likely to occur in large, complex or multicentre trials, where collusion between multiple parties would be required. Nevertheless, sometimes trials are encountered where it is suspected that the data have been purposefully manipulated or falsified, so such trials would need to be excluded from the analyses, and the reasons for doing so described in the meta‐analysis report.

Table 4.4Domains in the Risk of Bias 2 tool 91(RoB 2) of particular relevance to IPD meta‐analysis projects

Source: Based on Sterne JAC, Savovic J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 2019;366:l4898

Y: yes, PY: probably yes, N: no, PY: probably no, NI: no information. Note that in RoB 2, Domain 2 has an additional/alternative series of signalling questions for risk of bias arising from deviations from interventions in terms of adhering to interventions. The additional information that may be available from the IPD is similar to those given for the effect of assignment to the intervention in the table. Domain 5 has a sub‐list of questions that are not listed, because they are not relevant to IPD meta‐analysis projects.