Individual Participant Data Meta-Analysis

In the following sub‐sections, we suggest a range of checks for IPD obtained from randomised trials evaluating treatment effects, 7,9,43,101but most of these are applicable to other types of primary study.

When IPD are received for a trial, and often before processing the data further, it is worth conducting some preliminary checks. For example, it is useful to confirm that all the participants randomised appear to have been included, and check that there are no obvious omissions or duplicates in the sequence of participant identifiers (if they have been provided). Similarly, it is helpful to check which outcomes, baseline covariates and other variables are included in the IPD, and whether any that are ‘missing’ were truly not collected in the trial (called systematically missing variables ; Chapter 18) or were recorded, but not included in the IPD supplied. If the latter, then either more complete IPD should be requested again, or a full explanation for non‐provision sought.

For time‐to‐event outcomes, it is worth checking that the extent of follow‐up for each trial is sufficient for the condition and outcome of interest, and encouraging trial investigators to supply up‐to‐date follow‐up information where possible. 7Although short follow‐up may not introduce bias, it might prevent a trial, and therefore an IPD meta‐analysis, picking up benefits or harms of interventions that take a long time to accrue, such as late side effects of treatment or late recurrence of disease. For example, in an IPD meta‐analysis evaluating the addition of chemotherapy local treatment for soft tissue sarcoma, 102the median follow‐up for survival was reported for seven of the included trials as being between 16 and 64 months, 46which is rather short for this type of cancer. Trial investigators supplying IPD were asked to provide updated information, which extended the median follow‐up for these trials to between 74 and 204 months, 46and thus allowed a more reliable examination of the effects of chemotherapy in the long term ( Figure 4.5). 102Such updating may not be necessary if most or all events have already occurred, and may not be feasible if, for example, studies are very old, no longer obtaining follow‐up information, or resources are limited.

Figure 4.5Median follow‐up based on published aggregate data compared to updated obtained IPD for seven trials included in an IPD meta‐analysis of adjuvant chemotherapy for soft tissue sarcoma. 46

Source: Stewart et al. 46, © 2006, John Wiley & Sons.

While ideally these checks should make use of database or statistical code, the value of scanning the data by eye should not be underestimated, as it can help members of the central research team get a feel for the trial as a whole, and even highlight unusual patterns or peculiarities.

If data providers have followed the supplied data dictionary closely when preparing their IPD, much of the data harmonisation will have been done already, and minor adjustments may be all that are required. If trial investigators are unable or unwilling to prepare data according to suggested pre‐specified formats, the central research team should accept data in whichever format is most convenient, and recode it as necessary.

Beyond simply aligning trial IPD to the data dictionary, there is also the opportunity to standardise definitions of outcomes or participant‐level variables, 7,43such as scoring or staging systems. For example, in an IPD meta‐analysis examining the effects of chemotherapy for soft tissue sarcoma, 102different definitions of histological grade were used in the included trials, but with input from trial investigators, it was possible to translate each of these into a high‐ or low‐grade disease category, allowing exploration of treatment effectiveness according to grade. 43It may also be necessary to construct new standardised variables for use in analyses. For example, in an IPD meta‐analysis of the effects of antenatal diet and physical activity on maternal and foetal outcomes, the research team collected data on each woman’s height, baseline weight and parity, as well as the gestational age at birth and foetal birthweight for each baby. This allowed researchers to generate a standardised meta‐analysis definition of ‘small for gestational age’ (< 10th centile), using a bulk birthweight centile calculator. 103

IPD should be checked for invalid, outlying or implausible values, 7,9,43as these are the sorts of errors that can occasionally be missed during data input or checking of the original trials, or occasionally could arise as a result of variable recoding by the IPD meta‐analysis project team. Such checks might highlight, for example, records of unusually old or young patients, or those with abnormally high or low levels of important biomarkers, which need to be queried with trial teams. Of course, such apparently implausible values may turn out to be accurate. For example, in a trial included in an IPD meta‐analysis of neoadjuvant chemotherapy for cervical cancer, 104some invalid codes were used for the brachytherapy (internal radiotherapy) variable ( Figure 4.6), which were then queried and clarified as representing not applicable. When variables are related, checks should be carried out to ensure that they are coherent and consistent. For example, in a randomised trial, the recorded date of any event should follow the recorded date of randomisation and occur on or before the recorded date of last follow‐up.

At this stage, it is also useful to perform a simple descriptive analysis of the IPD from each trial to provide, for example, the number of participants, distribution of baseline characteristics by treatment group and overall results for the main outcome(s). These can then be checked for concordance with relevant publications or, for unpublished trials, with any results that have been deposited in trial registers (e.g. ClinicalTrials.gov). However, it should be borne in mind that inconsistencies can arise if, for example, follow‐up in a trial’s IPD has been extended beyond that used to derive the reported results, or if the meta‐analysis employs a different approach compared to the original trial analyses. When unexplained differences do arise, it is crucial to work with the original trial investigators to understand how and why they differ, and therefore, be in a position to report and explain any important discrepancies.

Figure 4.6Summary of the data validity, range and consistency checks on IPD from a trial included in an IPD meta‐analysis of neoadjuvant chemotherapy for cervical cancer 104(N.B. The participant references are pseudonymised).

Source: Based on Neoadjuvant Chemotherapy for Locally Advanced Cervical Cancer Meta-analysis C. Neoadjuvant chemotherapy for locally advanced cervical cancer: a systematic review and meta-analysis of individual patient data from 21 randomised trials. European journal of cancer 2003;39(17):2470–86.

Similar to conventional aggregate data reviews, assessing the reliability (quality) of included trials is also an important feature of the checking phase of IPD meta‐analysis projects. In such reviews, this is usually based on the risk of bias , a term that refers to the likelihood that included trials will generate biased results. In particular, the risk of bias assessment tool (RoB 2) can be used to evaluate potential bias in estimates of intervention effects from randomised trials. 91It includes five domains to be considered for each eligible trial: the randomisation process; deviations from intended interventions; missing outcome data; measurement of the outcome; and selection of the reported result. Within each domain, assessments are guided by multiple signalling questions (with answers: yes, probably yes, probably no, no, or no information), allowing a risk of bias classification for that domain (low, high, or some concerns). Finally, an overall risk of bias judgement can be made (low, high, or some concerns) based on all domains ( Section 4.7).