## 1 Introduction

Covariate adjustment is important in randomised controlled trials because it offers the promise of increasing the power to detect an effect of the randomised treatment (a treatment effect) [1]

. Covariate adjustment is usually done using a regression model including randomised treatment and a set of (preferably pre-specified) covariates. The regression model might be a linear model for quantitative data, a logistic regression for binary outcome data, or a linear mixed model or generalised linear mixed model for repeatedly measured data.

It is important to define which treatment effect is to be estimated, and we call this the estimand [2]

. Usually, the treatment effect is taken as the coefficient of randomised treatment in such models. In a linear model, covariate adjustment almost always leads to increased precision for this coefficient. However, in non-linear models this is often not the case, and covariate adjustment instead increases the standard errors

[3]. This paradox is explained by the fact that covariate adjustment changes the estimand [4]; when coefficients are compared on the same scale, covariate adjustment does increase their precision. Further, covariate adjustment increases power in all cases [5].Sometimes, the treatment effect is instead derived from the fitted model. For example, the treatment effect on a binary outcome may be defined as the risk difference between experimental and control arms. This may be derived from the fitted model by standardisation, which predicts the outcome for each randomised individual’s covariate pattern under each possible randomised treatment and takes averages over individuals [4]. The treatment effect may also be derived by inverse probability of treatment weighting, which uses covariates to predict randomised treatment and then uses the fitted inverse probability of each individual having received their actual treatment as a set of weights that induces covariate balance in a subsequent unadjusted analysis [6].

A common concern is that the more complex models involved in covariate adjustment are more open to model mis-specification and therefore to bias in the treatment effect [7]. Two main forms of model mis-specification are possible. Mis-specification of the main effects of the covariates, or omissions of covariate-covariate interactions, causes a failure to realise the full potential gain in power, but is unlikely to cause bias. The omission of randomised treatment by covariate interactions, by contrast, has the potential to cause bias.

We focus on analysis of randomised trials using generalised linear models (GLMs) [8]

. These regression models are defined by an outcome distribution and a link function. For each outcome distribution there is one link function, called the canonical link, which simplifies the form of the score equations. Most widely used GLMs have canonical link functions, e.g. linear regression (Normal distribution and identity link), logistic regression (binomial distribution and logit link) and Poisson regression (Poisson distribution and log link). However non-canonical links are occasionally used, and in particular the model with a binomial distribution and identity link can be used to estimate a covariate-adjusted risk difference.

Other work has focussed on large-sample properties of covariate adjustment with mis-specified models. For example, even with mis-specified models, estimation of a marginal treatment effect using a generalised linear model (GLM) with canonical link is consistent [9]; a wide range of hypothesis tests, based on GLMs with canonical links, asymptotically retain correct type 1 error [10]; and a standardisation approach using a canonical-link GLM within each trial arm is asymptotically unbiased and locally efficient [7]. However, large-sample properties are of limited value when assessing the results of a single moderate-sized trial.

This short note focuses on trials of any size and shows that a large degree of protection against model mis-specification is conferred by using generalised linear models with canonical link functions, and conversely that using non-canonical link functions leads to potentially important error.

## 2 Example

Table 1 shows hypothetical data from a two-arm randomised trial, where the binary outcome represents an unfavourable outcome. These data show that the experimental randomised treatment reduces the risk in the subgroup but increases the risk in the subgroup, in such a way that the overall effect of randomised treatment is zero. That is, if this treatment were given to a population which (like the trial sample) consisted of equally sized and subgroups, then the overall change in risk would be zero.

In practice, analysis of these data might identify this interaction (it is just statistically significant, P=0.03). However, if the interaction were unexpected and not specified in the statistical analysis plan, then analysis would probably proceed to estimate the overall treatment effect in all patients.

Subgroup | (Experimental) | (Control) | Risk difference (E-C) |
---|---|---|---|

10/200 (5%) | 20/200 (10%) | -5% | |

90/200 (45%) | 80/200 (40%) | 5% | |

All | 100/400 (25%) | 100/400 (25%) | 0% |

We analysed these data using three link functions: logistic, identity and probit, and with and without adjustment for the covariate . The estimand of interest is the treatment effect expressed as a risk difference. We estimated this treatment effect as the coefficient of in the identity link model. In the models with other links, we estimated the treatment effect using standardisation (using Stata’s margins command [11]).

The results in Table 2 show that, as expected, all unadjusted models report a treatment effect of zero with very similar standard errors. Among the adjusted analyses, only the logistic analysis reports a treatment effect of zero. Thus despite the model mis-specification, the logistic approach is correct. This robustness is not seen for the non-canonical links, with the adjusted identity link model in particular wrongly estimating that randomised treatment reduces risk by 2.8%. These results are not a small-sample properties, since multiplying all counts in Table 1 by a large number would not change the point estimates. The error with a non-canonical link therefore represents both bias and inconsistency.

Link | Unadjusted | Adjusted | ||
---|---|---|---|---|

Coefficient | Standard | Coefficient | Standard | |

error | error | |||

Logit (canonical) | 0.000 | 0.031 | 0.000 | 0.028 |

Identity (non-canonical) | 0.000 | 0.031 | -0.028 | 0.023 |

Probit (non-canonical) | 0.000 | 0.031 | -0.006 | 0.028 |

## 3 Mathematical exploration

This section shows that if (1) a GLM with canonical link is used and (2) the covariate is perfectly balanced across randomised treatments , then the coefficient of the being zero in the unadjusted model implies that the coefficient of is also zero in the unadjusted model. We assume a two-arm trial, but the argument generalises to more arms.

We write the unadjusted and adjusted analysis models as

(1) | |||||

(2) |

where is the inverse link function and is the outcome.

The score equations for the unadjusted and adjusted models with canonical link are

(3) | |||||

(4) |

where are the values of for the th individual, . However, rather than estimating , we will show that the result holds for any given value of (and hence that it holds for the value ). By fixing we only need to solve the first two components of equation (4).

We assume that the unadjusted treatment effect is zero: that is, equation (3) has a solution with . This means that

(5) |

We then show that whatever the value of in model (2), the value solves the first two components of equation (4). The assumption of perfect balance implies that the average of in the subgroup is the same as its average across all individuals. Let this average be . We can therefore write the adjusted model score as

(6) |

This shows that values satisfying and solve equation (3). These values exist for any link function that is continuous and monotonic. Hence the adjusted treatment effect is also zero.

The converse is also easy to show, that if the adjusted treatment effect is zero then the unadjusted treatment effect is also zero.

If instead a non-canonical link were used then the summands in equations (3) and (4) would be multiplied by a further factor equal to the derivative of the canonical parameter with respect to the linear predictor, and this would depend on the covariate . The above argument would therefore not hold. For example for a binary outcome with probability , the canonical parameter is . If logistic regression is used then the linear predictor is also and equations (3) and (4) are correct. If a GLM with identity link is used then the linear predictor is and the adjusted model score equation becomes

(7) |

## 4 Systematic exploration

We explored data sets like that in Section 2, with the same marginal distribution of and , but varying the numbers of positive outcomes in each cell from 10 to 20 in steps of 2. This yielded data sets. We analysed each data set using identity, log and logit links, unadjusted and adjusted for the covariate . To explore the impact of adjustment on the model coefficients, we did not perform the standardisation step in Section 2: coefficients are therefore not comparable between the different link functions. Unadjusted and adjusted coefficients are also not comparable for the logit link, though the latter difference tends to be small.

Figure 1 shows Bland-Altman plots [12] comparing the unadjusted and adjusted estimates of the treatment effect, for the three link functions. The top panel, with the canonical link function, shows a pattern of the adjusted estimator being less than the unadjusted when the average is below zero, and more than the unadjusted when the average is above zero. This implies that the adjusted estimator is always more extreme (further from the null) than the unadjusted estimator. The middle and bottom panels, with non-canonical links, shows a very different and more diffuse pattern. In particular, there is a band of points with an average coefficient zero, implying that the unadjusted and adjusted coefficients may have different signs, even though the covariate is perfectly balanced across randomised treatments. This would be a worrying finding in the analysis of a randomised trial.

## 5 Discussion

We have shown that canonical link functions provide special protection against error due to interactions when generalised linear models are fitted to randomised trial data.

It could be argued that the magnitudes of error that we have shown with non-canonical links – for example, 3 percentage points on the risk difference – are unimportant. The error would be larger if the interaction were larger. Here it could be argued that large interactions would always be detected. However, there is an intermediate area where interactions are not statistically convincing and error remains of an important magnitude. Further, in small data sets, even large interactions may be not statistically significant and hence may not be believed, and here error in covariate adjustment without a canonical link could be large. We have not done a simulation study to demonstrate bias across many data sets, but nevertheless we have shown that these errors would arise in large samples and do constitute bias.

Fortunately there is a simple way to avoid bias, and that is to use canonical links. This means that for binary outcome data, estimation of the risk difference with covariate adjustment should not be done by directly fitting an identity-link binomial regression model, which has a non-canonical link. A good alternative is to model the data using a canonical-link GLM and performing post-estimation standardisation, as we did in Section 2. Another alternative is to use inverse probability of treatment weighting in a simple canonical-link GLM. A separate paper more fully describes these approaches and their advantages and disadvantages [13].

We have focussed on the case of perfect covariate balance. This has allowed us to focus on evaluating adjusted estimates when the unadjusted estimate is zero. Perfect covariate balance can realistically be achieved in trials with a small number of categorical covariates, provided the sample size is an exact multiple of the randomised block length, but it cannot be achieved in trials with continuous covariates. Our findings about the superiority of canonical links are likely to hold in trials that do not have perfect covariate balance, and this could be explored using simulation. However, some bias may still occur with canonical links. In other unpublished work, we have observed that covariate adjustment with a severely mis-specified model and a continuous covariate can led to small-sample bias, though not to large-sample bias.

The implications of this work for analysis of time-to-event outcomes are unclear, since such outcomes are usually analysed by Cox models which are not in the GLM family. It could be that the close link of Cox models with canonical-link Poisson regression provides reassurance of robustness against model mis-specification.

## 6 Conclusions

We conclude that non-canonical links should be avoided for covariate adjustment in randomised trials. If a marginal risk difference is the target of estimation, then this should be estimated using logistic regression with standardisation or inverse probability of treatment weighting.

## Acknowledgements

IRW and TPM were supported by the Medical Research Council Unit Programme number MC_UU_12023/21. EJW was supported by the MRC Network of Hubs for Trials Methodology HTMR Award MR/L004933/2/N96 and MRC project grant MR/S01442X/1. We thank Brennan Kahan and Tim Clark for the encouragement to write this work up and Kelly van Lancker for helpful advice.

## References

- [1] Kahan BC, Jairath V, Doré CJ, Morris TP. The risks and rewards of covariate adjustment in randomized trials: an assessment of 12 outcomes from 8 studies. Trials 2014; 15(1): 139.
- [2] International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use; 2019. Addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials E9(R1). https://www.ich.org/page/efficacy-guidelines#9-2.
- [3] Robinson LD, Jewell NP. Some surprising results about covariate adjustment in logistic regression models. International Statistical Review / Revue Internationale de Statistique 1991; 59(2): 227.
- [4] Daniel R, Zhang J, Farewell D. Making apples from oranges: Comparing noncollapsible effect estimators and their standard errors after adjustment for different covariate sets. Biometrical Journal 2021; 63(3): 528–557.
- [5] Neuhaus JM. Estimation efficiency with omitted covariates in generalized linear models. Journal of the American Statistical Association 1998; 93(443): 1124–1129.
- [6] Williamson EJ, Forbes A, White IR. Variance reduction in randomised trials by inverse probability weighting using the propensity score. Statistics in Medicine 2014; 33(5): 721–737.
- [7] Rosenblum M, van der Laan MJ. Simple, efficient estimators of treatment effects in randomized trials using generalized linear models to leverage baseline variables. The International Journal of Biostatistics 2010; 6(1): 13.
- [8] McCullagh P, Nelder JA. Generalized Linear Models. Chapman and Hall; London; 1989; 2nd edition.
- [9] Bartlett JW. Covariate adjustment and estimation of mean response in randomised trials. Pharmaceutical statistics 2018; 17(5): 648–666.
- [10] Rosenblum M, van der Laan MJ. Using regression models to analyze randomized trials: asymptotically valid hypothesis tests despite incorrectly specified models. Biometrics 2009; 65(3): 937–945.
- [11] StataCorp. Stata Statistical Software: Release 11. Stata Press; College Station, TX; 2009.
- [12] Bland JM, Altman DG. Statistical methods for assessing agreement between 2 methods of clinical measurement. Lancet 1986; 327: 307–310.
- [13] Morris TP, Walker AS, Williamson EJ, White IR. Planning a method for covariate adjustment in individually randomised trials: a practical guide. arXiv 2021; 2107.06398.

Comments

There are no comments yet.