Individual Participant Data Meta-Analysis

Trials that stop early can produce results that are overly in favour of treatment or control, 117and therefore introduce bias into a subsequent meta‐analysis. For those based on time‐to‐event outcomes, obtaining IPD with updated follow‐up can go some way to addressing this issue. For example, in an IPD meta‐analysis examining adjuvant chemotherapy for locally advanced bladder cancer, three of the included trials were stopped early, because they had interim results in favour of adjuvant treatment. However, the IPD meta‐analysis project helped alleviate this potential bias, as it included IPD with updated follow‐up for the three trials, which produced results that were less in favour of adjuvant treatment ( Table 4.5). 118IPD also allows non‐proportional hazards (non‐constant hazard ratios) to be examined (see Chapter 5).

Figure 4.12‘Reverse’ Kaplan‐Meier analysis of participants who are event‐free for (a) a trial with balanced follow‐up, and (b) a trial with imbalanced follow‐up that was (c) subsequently updated with longer follow‐up when IPD were collected. Each was included in an IPD meta‐analysis of adjuvant chemotherapy for soft tissue sarcoma. 102

Source : Sarcoma Meta-Analysis Collaboration. Adjuvant chemotherapy for localised resectable soft-tissue sarcoma of adults: meta-analysis of individual data. Lancet 1997;350(9092):1647–54.

Table 4.5Alleviating potential bias in trials that stopped early for perceived benefit (included in an IPD meta‐analysis of adjuvant chemotherapy for locally advanced bladder cancer) through updated follow‐up

Source: Jayne Tierney, adapted with permission. 118

The results of validity checking ( Section 4.5) and risk of bias assessment ( Section 4.6) should be considered together in order to build up an overall picture of the quality of each trial’s IPD. This should include reflections on the quality of the trial design and conduct (from the ROB 2 assessment), checks of IPD obtained, and any unresolved errors or concerns therein. If it is concluded that the IPD from a particular trial is likely to introduce considerable bias into an IPD meta‐analysis, then it is may be sensible to exclude it. For example, in an IPD meta‐analysis of post‐operative therapy for non‐small‐cell lung cancer, 107a trial was excluded because it ‘failed’ the data checks, 101and it is certainly worth highlighting any such exclusions in the relevant meta‐analysis publication. However, such situations need to be handled sensitively with trial investigators, who will have invested time and effort in supplying the data, and may have been unaware that issues would emerge. Alternatively, the impact of risk of bias may be explored through sensitivity analysis, such as examining how meta‐analysis conclusions change according to whether or not trials have risk of bias concerns ( Chapter 9).

The ‘traffic light’ table used for standard risk of bias assessment can be usefully adapted for summarising the overall quality of trials included in an IPD meta‐analysis. This would include all the domains described previously, except the “selection of the reported results” domain, which is not applicable to IPD projects ( Section 4.6), because the trial IPD are re‐analysed according to the meta‐analysis protocol and SAP. Adopting this structure means that the information is readily comparable with a standard risk of bias table, but can easily be extended to incorporate columns for additional project‐specific IPD checks that are deemed particularly important. For example, it is useful to include an additional column to indicate whether there were any residual concerns about data quality once the data checking and correction procedures had been completed, and which might impact on the trustworthiness of a trial. Often these risk of bias ‘traffic light’ tables will be almost completely green (low risk of bias), as any trials that fail data checking or have serious bias issues would likely be excluded from the IPD meta‐analysis project completely. Note that assessments should be based on the fullest information possible, thereby considering the trial design and conduct based on all trial documentation and contact with investigators, plus the results of checks of the IPD. A more detailed risk of bias table might be included in an appendix to provide fuller information on this data checking process ( Table 4.6), and show how individual judgements have been arrived at. It also provides an opportunity to flag less serious or unclear bias issues.

Table 4.6Excerpt of a RoB2 table for an IPD meta‐analysis of adjuvant chemotherapy for locally advanced bladder cancer based on a single trial and the main outcome of overall survival (Tierney et al., in preparation).

Source: Sarah Burdett and Jayne Tierney.