Individual Participant Data Meta-Analysis

Details of ways to evaluate the IPD in relation to the relevant RoB 2 domains are provided in the following sections. Findings of risk of bias and data checking can usefully be presented using a modified version of the ‘traffic light’ table used for standard risk of bias assessments ( Section 4.7).

For randomised trials, it is important to check that the methods of sequence generation and allocation concealment appear robust. This will help guard against the inclusion of non‐randomised trials, or non‐randomised participants, in the IPD meta‐analysis project. While a description of the methods can be gleaned from trial documentation and/or trial personnel, interrogating the IPD directly can highlight any unusual allocation patterns that may need further investigation. 7,9,43

A simple method of checking the pattern of allocation is to look at the cumulative number of participants randomised into each treatment group over time. For example, Figure 4.7shows the pattern for a trial included in an IPD meta‐analysis evaluating the effects of chemoradiation for cervical cancer. 93There are a similar number of participants allocated to each group throughout, and the curves cross frequently, which is what we would expect in a trial with robust randomisation procedures. It should be noted that for smaller trials, greater separation of curves and less crossing over might be expected, even if they are properly randomised, particularly if a simple method of randomisation was used. Also, where the allocation ratio is not 1:1, the curves would not be expected to cross, but rather would be expected to track one another.

Figure 4.7The cumulative number of participants randomised to the intervention and control groups in a trial included in an IPD meta‐analysis of chemoradiation for cervical cancer. 93

Source: Based on Chemoradiotherapy for Cervical Cancer Meta-Analysis Collaboration. Reducing uncertainties about the effects of chemoradiotherapy for cervical cancer: a systematic review and meta-analysis of individual patient data from 18 randomized trials. Journal of clinical oncology: official journal of the American Society of Clinical Oncology 2008;26(35):5802–12.

A similar pattern is seen in the early stages of a trial included in an IPD meta‐analysis of chemotherapy versus radiotherapy for multiple myeloma, but for a short time period all the participants were allocated to the chemotherapy group ( Figure 4.8). 7The trial investigator explained that this was when the radiotherapy machine had broken down, and all participants were given chemotherapy. As this particular cohort of participants were not randomly allocated to treatment, they were excluded from the IPD, thereby still allowing the trial to be included in the IPD meta‐analysis.

Figure 4.9shows another example of how treatment allocation can be plotted, and was used to examine the pattern of assignment in a trial from another IPD meta‐analysis project. In this case, all participants except one were allocated to one intervention (arm 1) in the first months of the trial, and participants were allocated to the other intervention (arm 0) mostly in the final months. The trial investigator was unable to explain the pattern, agreed that it was not consistent with randomisation, and that the trial should be excluded from the meta‐analysis. As this was a single‐centre trial, where treatments were supplied in independently pre‐prepared trial packs (with no indication of the treatment they contained), it was speculated that these had not been mixed sufficiently.

Figure 4.8The cumulative number of participants allocated to chemotherapy or radiotherapy in a trial included in an IPD meta‐analysis of treatments for multiple myeloma.

Source: Stewart et al., 7. © 1995, John Wiley & Sons.

Figure 4.9Date (shown by year‐month) participants were allocated to treatment and control in a trial excluded from an IPD meta‐analysis, because participants in one group (‘arm 1’) were generally recruited earlier than those in the other group (‘arm 0’).

Source: Lesley Stewart.

As statistical software and database packages allow dates to be converted to days of the week, another simple check is to look at the number of participants allocated to the research and control groups on each day of the week. 7,106This method would highlight, for example, if participants were being allocated to treatment on particular clinic days (pseudo‐random allocation) or if participants were being allocated on the weekends, which would be unusual for trials in chronic conditions in many countries. For example, in a trial included in an IPD meta‐analysis examining pre‐operative chemotherapy for lung cancer, 88there were no weekend randomisations, and the numbers randomised to each treatment group were well balanced on the weekdays ( Figure 4.10(a)). In contrast, for a trial included in an IPD meta‐analysis examining post‐operative radiotherapy for lung cancer, 107there appeared to be an unusually high number of weekend randomisations, and large imbalances in the number of participants allocated to each group on each day ( Figure 4.10(b)). When this was brought to the attention of the trial investigator, they discovered problems with the management of the trial data, and went back to individual participant records, to ensure that the appropriate information was supplied, and the issues were resolved.

As mentioned in Section 4.4.1, if the dates of randomisation have been redacted from trial IPD for de‐identification purposes, it will not be feasible for the central research team to employ these checking procedures, but the trial statistician may be able to run these on their behalf. Moreover, it is still possible to visually check whether baseline characteristics appear reasonably balanced by group, as we would expect with a robust randomisation process. However, balance will never be perfect, and imbalances may be more pronounced in small trials or those with simple (i.e. non‐stratified) randomisation methods; indeed we might be concerned if everything appeared too perfectly balanced. Note that we do not advocate statistical tests of baseline balance. 108

While robust randomisation procedures should ensure the unbiased assignment to, and comparison of, participants between treatment groups, this can only be guaranteed if all participants are analysed according to the treatments initially assigned: an intention‐to‐treat approach. 109–111Even if a trial has not been analysed appropriately, as long as the IPD have been provided with the original treatment allocation recorded, then participants can be grouped according to this treatment allocation, rather than the treatment they received, enabling an intention‐to‐treat analysis of effectiveness. 109–111However, there may be value in conducting certain analyses based on a subset of participants randomised, such as an analysis of toxicity in just those who received most of their allocated treatment, or sensitivity analyses according to treatment received, to explain differences between published trial results and those used in the meta‐analysis.