Iain K. Crombie - Evidence in Medicine

The impact of poor quality blinding on estimates of treatment effect is unclear, as review studies give conflicting results. Two overview studies found that poor blinding was associated with an increased effect size compared to well‐blinded trials [7, 32]. Another study found that this bias only occurred for subjective outcome measures [2], and a fourth reported an inconsistent effect [18]. The most recent, and largest, study concluded that there was no effect of blinding [33].

A basic principle of pharmacology is that ‘all drugs have beneficial and harmful effects’ [34], with the value of the drug depending on the benefit: harm ratio. Establishing where the balance lies can be difficult because the harms are frequently under‐reported in published clinical trials [35, 36]. One study found that 43% of adverse events recorded in trial registries were not reported in the published study [37]. A review of such studies concluded that, on average, some 64% of harms were not reported [38]. The benefit: harm ratio may often be biased because of the under‐reporting of harms [39].

In a trial, patients are followed up for a defined period to determine the effect of treatment. During this time period some patients can withdraw from the study, and contact can be lost with others, so that the outcome is often not measured on all patients. A rule of thumb for losses to follow‐up is that <5% loss is unlikely to cause bias and that >20% loss is serious, with those between 5% and 20% being potentially problematic [40, 41].

Loss to follow‐up occurs in almost all clinical trials, although reporting of this is often incomplete. One problem is that some published papers give no information about loss to follow‐up. A study of trials in five leading journals found that 13% failed to report whether or not loss to follow‐up had occurred [42]. Other studies have found that the lack of an explicit statement about missing data occurred in 6.5% [43] and 26% [44] of published papers.

Trials that report the extent of loss to follow‐up vary greatly in the proportion of participants affected. Among 77 trials published in leading medical journals in 2013, 95% reported some missing outcome data: although the median loss was 9%, the highest reported was 70% [45]. Similarly in trials funded by a Health Technology Assessment programme, the median loss was 11%, but this ranged from 0% to 77% in individual studies [46].

In many trials loss follow‐up exceeds the notional threshold of 20% for serious loss. The extent of this may vary across specialties. Among trials in palliative care it was 23% [47], slightly higher in osteoarthritis (26%) [44] and 39% in rheumatoid arthritis [48]. Among obesity trials 74% reported losses >20% [49].

Care is needed in the interpretation of the loss to follow‐up rates. In a study of 21 trials, the data reported to the US Food and Drugs Administration showed markedly higher median loss to follow‐up (13%), than that in the corresponding published papers (0.3%) [50]. The authors of this study concluded that the ‘published rates consistently seem to be inadequate representations of the completeness and quality of follow‐up’.

The type of patient lost to follow‐up is possibly more important than the number lost. If more seriously ill patients are lost, and this loss happens to a greater extent in one of the treatment groups, then substantial bias can occur. Thus when loss to follow‐up occurs, trials need to report not just the magnitude of loss, but the number and characteristics of the patients lost from each of the treatment arms.

Trials that report the overall loss to follow‐up often do not give the data separately for intervention and control groups. One study of trials in leading medical journals found that 20% did not report the numbers missing in the treatment and control groups separately [42]. However a more recent study found that leading journals had improved, with only 3% of trials failing to report this information [45]. Another study of trials in palliative care found that 13% did not report the numbers lost to follow‐up in both treatment arms [51]. In these studies it is not clear whether treatment effects will be biased by differential loss to follow‐up.

Reporting of the types of patients lost to follow‐up is often poor. In one review, 91% of trials did not compare the characteristics of those lost to follow‐up with those successfully followed up [43]. Among 108 trials in palliative care, none compared the intervention and control groups for the baseline characteristics of those with missing outcome data [47]. Thus it is often not possible to assess whether those lost to follow‐up in the intervention group differ from those lost in the control group.

The impact of loss to follow‐up is difficult to predict. A review found that loss to follow‐up was sometimes higher in the treatment group and sometimes in the comparator [47]. Possibly the effect of loss to follow‐up depends on the specific trial characteristics, such as the illness being treated or the acceptability of the treatment to patients. Whatever the explanation, concern remains that some trials may be biased due to patient attrition.

A difficult issue for the analysis of a trial is how to cope with those who are lost to follow‐up. The accepted approach to this problem is termed ‘intention to treat’. This holds that all patients should be analysed in the groups to which they were randomised, no matter what subsequently happened to them. Intention to treat is regarded as ‘a key defence against bias’ in clinical trials [52].

A popular technique to handle missing data is complete case analysis, which restricts the analysis to those successfully followed up. This approach was used in 26% of trials on musculoskeletal disorders [53], in 45%–54% of studies in general medicine [45, 54] and in 60% of trials in palliative care [51]. Despite its popularity, complete case analysis clearly contravenes the principle of intention to treat, effectively losing the benefit of randomisation. This method is likely to introduce bias, because the number and types of patients lost to follow‐up are unlikely to be the same in the intervention and control groups [55].

Instead of ignoring the problem, as complete case analysis does, a better approach is to use a method to estimate what the missing value might be. This is termed imputation. The simplest form of imputation is last observation carried forward. If trials measure the outcome at more than one time point, the most recent observation is used in the analysis; if there are no intermediate measures the baseline measurement is used. This method is often used in trials [45, 48], but it has been heavily criticised because it is likely to lead to biased estimates of treatment effects [55–57]. As disease severity often changes over time, with relapses or remissions, early values can be poor predictors of the final outcome.

Multiple imputation is a more sophisticated approach to estimate the missing outcome data. It uses statistical modelling to derive estimates of the missing data based on the available data (imputation). The modelling includes the variables that would normally be used in the final analysis, such as the stratification variables (e.g. centre, gender), treatment group and potential confounders (e.g. initial disease severity, other medical conditions). It can also include other variables (auxiliary variables) that might be available [58, 59]. The idea is that patient characteristics, both medical and demographic, could predict the final outcome. The modelling process is repeated many times to produce an average result (hence multiple imputation).