Estimating Individualized Treatment Regimes from Crossover Designs

02/05/2019 ∙ by Crystal T. Nguyen, et al. ∙ 0

The field of precision medicine aims to tailor treatment based on patient-specific factors in a reproducible way. To this end, estimating an optimal individualized treatment regime (ITR) that recommends treatment decisions based on patient characteristics to maximize the mean of a pre-specified outcome is of particular interest. Several methods have been proposed for estimating an optimal ITR from clinical trial data in the parallel group setting where each subject is randomized to a single intervention. However, little work has been done in the area of estimating the optimal ITR from crossover study designs. Such designs naturally lend themselves to precision medicine, because they allow for observing the response to multiple treatments for each patient. In this paper, we introduce a method for estimating the optimal ITR using data from a 2x2 crossover study with or without carryover effects. The proposed method is similar to policy search methods such as outcome weighted learning; however, we take advantage of the crossover design by using the difference in responses under each treatment as the observed reward. We establish Fisher and global consistency, present numerical experiments, and analyze data from a feeding trial to demonstrate the improved performance of the proposed method compared to standard methods for a parallel study design.



There are no comments yet.


This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Personalized medicine is the practice of tailoring treatment to account for patient heterogeneity (Chakraborty and Murphy, 2014). Physicians and other health care providers have practiced personalized medicine by adjusting doses or prescriptions based on a patient’s medical history or demographics for centuries (Ashley, 2015; Zhao and Zeng, 2013). Precision medicine is an emerging field that aims to support personalized medicine decisions with reproducible research (Collins and Varmus, 2015). Such research is imperative, particularly when diseases are expressed with great heterogeneity across patients. A topic of interest in precision medicine is the individualized treatment regime (ITR): a set of decision rules for one or more decision time points that can be used to assign patients to treatment tailored by their patient-specifict factors (Lavori and Dawson, 2014; Moodie et al., 2007; Petersen et al., 2007). One objective in precision medicine is to estimate the optimal ITR, or the ITR that maximizes the mean of some desirable outcome (Kosorok and Moodie, 2015; Laber et al., 2014). Crossover clinical trials are uniquely suited to precision medicine, because they allow for observing responses to multiple treatments for each patient. This paper introduces a method to estimate optimal ITRs using data from a crossover study by extending generalized outcome weighted learning (GOWL) (Chen et al., 2018) to deal with correlated outcomes.

In a crossover study, patients are randomized to a sequence of treatments rather than a single treatment. Thus, multiple outcomes are observed, one per subject from each treatment period, and each subject acts as his or her own control for reduced between-subject variability (Machin and Fayers, 2010; Turner, 2010; Wellek and Blettner, 2012). Therefore, crossover designs naturally lend themselves to precision medicine; estimating the optimal ITR from a crossover design can utilize all counterfactual outcomes. In contrast, estimating the optimal ITR from traditional parallel group designs, where patients are assigned to a single treatment, can only utilize the subset of counterfactual outcomes that are observed.

There have been many developments in machine learning methods for answering precision medicine questions from parallel study designs. For example, Qian and Murphy (2011)

indirectly estimate the decision rule using L1 penalized least squares; Zhang et al. (2012a) maximize a doubly robust augmented inverse probability weighted estimator for the population mean outcome;

Athey and Wager (2017) maximize a doubly robust score that may take into account instrumental variables; Kallus (2018) employs a weighting algorithm similar to inverse probability weighting but minimize the worst case mean square error; Laber and Zhao (2015)

propose the use of decision trees, which prove to be both flexible and easily interpretable;

Zhao et al. (2012), Zhang et al. (2012b), Zhou et al. (2017), and Chen et al. (2018) directly estimate the decision rule by viewing the problem from a weighted classification standpoint.

However, little work has been done to develop precision medicine methods that handle correlated observations in the single-stage decision setting such as those that arise from crossover designs. Kulasekera and Siriwardhana (2018) propose a weighted ranking algorithm to estimate a decision rule that maximizes either the expected outcome or the probability of selecting the best treatment, but they assume that there are no carryover effects present. Because the intended effect of the washout period can be difficult to achieve in practice (Wellek and Blettner, 2012), it is imperative that methods for crossover designs can be applied when carryover effects are present. In this paper, we show that the difference in response to two treatments from a crossover trial can be used as the reward in the generalized outcome weighted learning (GOWL) objective function to estimate an optimal ITR. We introduce a plug-in estimator that can be used with the proposed method to account for carryover effects. Additionally, we show that using a crossover design with the proposed method results in improvements in misclassification rate and estimated value when compared to standard methods for a parallel design at the same sample size.

As a clinical example, consider nutritional recommendations surrounding the intake of dietary fiber for the purpose of weight loss. Although increased fiber is recommended across the population for a myriad of health benefits (Anderson et al., 1994; Anderson et al., 2009; Marlett et al., 2002; US Department of Agriculture, 2010), evidence of the impact of the consumption of dietary fiber for improved satiety and reduction in body weight is mixed (Halliday et al., 2018; Slavin, 2005). Heterogeneity in response to dietary fiber may be leveraged to develop targeted fiber interventions to promote feelings of satiety. We use data from a crossover study in which Hispanic and African American adolescents who are overweight and obese were fed breakfast and lunch under a typical western high sugar diet and a high fiber diet. From these data, we estimate a decision rule with which clinical care providers can input patient characteristics, including demographics and clinical measures, and receive a recommendation to maximize the change in measures of perceived satiety from before breakfast to after lunch. This type of analysis could be useful in identifying a subgroup of at-risk adolescents for which targeting specific dietary recommendations is expected to lead to an increase in patient-reported satiety, helping to decrease caloric intake in a population with great clinical need for effective weight loss strategies.

The rest of this paper is organized as follows. In Section 2, we review outcome weighted learning (OWL) (Zhao et al., 2012) and present the proposed method for estimating an optimal ITR from a crossover study regardless of the presence of carryover effects. Section 3 establishes Fisher and global consistency. Section 4 demonstrates the performance of the proposed method in simulation studies, with results on misclassification rate and estimated value. Section 5 reports on the analysis of data from a feeding trial with overweight and obese Latino and African American adolescents, and we conclude with a discussion in Section 6.

2 Methodology

In this section, we provide a brief overview of existing methods for estimating the optimal ITR using weighted classification. We then provide the justification and means to implement our proposed method, which we will from here refer to as “crossover GOWL.”

2.1 Existing Methods

Consider a parallel, two-arm clinical trial in which we have i.i.d. observations for , where is binary treatment assignment, is a

-dimensional vector of covariates, and

is a reward, bounded by , for which greater values are desired. Assume that is of the form

where is the main effect of the covariates, is the treatment-covariate interaction, and

has mean 0 and variance

. Denote as the counterfactual outcome under treatment . We then make three causal assumptions (Rubin, 1978) to connect the counterfactual outcomes to the observed data: with probability 1, , and . These are known as positivity, conditional exchangeability, and consistency, respectively.

An ITR, , comes from the set of all functions that map the covariate space to the treatment space . Our objective is to estimate the optimal ITR, denoted , which maximizes the value function (Qian and Murphy, 2011),


where is the propensity score for treatment. Equivalently, may be defined as


Zhao et al. (2012) propose OWL to solve this problem: each misclassified observation is weighted by its observed outcome,

, and the hinge loss is used to bring the problem into the support vector machine framework

(Cortes and Vapnik, 1995). Unfortunately, OWL assumes is nonnegative; when negative values are observed, OWL shifts all outcomes to be nonnegative, since (2) is invariant to such a transformation. The objective function in OWL, however, does not have this property. Therefore, the estimated decision function in OWL depends on the chosen shift in the outcomes. Chen et al. (2018) propose GOWL, an extension of OWL, which handles negative rewards by modifying the hinge loss to be piecewise and weighting the misclassified observations by . With GOWL, there is no need to shift rewards.

However, neither Zhao et al. (2012) nor Chen et al. (2018) considered correlated outcomes, such as those that arise from a crossover design setting. We now introduce crossover GOWL, a method that combines the observed treatment response difference with GOWL to estimate the optimal ITR from crossover data.

2.2 Crossover Generalized Outcome Weighted Learning

In a crossover design, patients are randomly assigned to a sequence of treatments rather than a single treatment. For the design, patients are randomized to receive either the or the sequence, with some prespecified washout period between treatments. The washout period is a break between treatments which serves to remove any carryover effects, or residual effects remaining from a previous treatment at the start of the next treatment. Keeping most of the notation from before, we now introduce sequential treatments and outcomes and for periods , respectively, i.e., is the observed outcome after receiving treatment in period . Furthermore, we assume the model

where has mean and a positive definite covariance matrix, , and is the carryover effect which may depend on and . Note that in a crossover study, the period effects, or temporal effects, are nonseparable from the carryover effects (Fleiss, 1989), so encompasses both period and carryover effects.

Let . Given the observed data , we propose the following as a substitute for the value function to be maximized:


where is the probability of being assigned to the sequence conditional on . Under Lemma 2.1, maximizing (3) is equivalent to maximizing (1); the proof is left to Appendix C.

Lemma 2.1.

Under the given assumptions,

Following (3), we use an approach similar to GOWL but weight misclassified observations by the treatment response difference, and we minimize the objective function (4) for in , a class of functions, e.g., a reproducing kernel Hilbert space. Let , be a tuning parameter, and be the norm of . For details on solving the minimization problem in (4), we defer to Chen et al. (2018) and Kimeldorf and Wahba (1970).


In practice, the true value of

is unknown. In traditional analyses, we are concerned with testing the null hypothesis that

. Here, we are instead interested in whether or not either treatment has a nonzero carryover effect. Investigators may determine whether carryover effects are present any number of ways, including two-sample -tests for the null hypotheses and by comparing mean responses to each treatment at each time point. An estimator for , denoted , can be computed using Algorithm 1.

1Estimate by regressing on and .
2 Set .
Estimate by regressing on and .
Algorithm 1 Estimating

In short, one model is fit to predict what would have been observed in period 2 in the absence of carryover effects, and another model is fit to predict the residual from the first model. While any regression technique may be used here, we use reinforcement learning trees (RLT) in our implementation. RLT is a nonparametric tree-based machine learning method that considers future splits or branches in the model when determining the best split at any node

(Zhu et al., 2015).

We can now correct the observed reward with the estimated carryover effects. Letting the estimated decision function is


and our proposed estimator of the optimal ITR is , where

3 Theoretical Results

In this section, we establish both Fisher and global consistency. First, define the risk under 0-1 loss to be

The risk under the modified loss function with the reward defined as the treatment response difference is then

Let , so that the corresponding ITR under the modified loss for the treatment response difference is . Under Theorem 1, Fisher consistency for is derived.

Theorem 3.1.

Under the given assumptions, .

Consider that for some kernel function , and let be the closure of . Define to be the minimizer over all functions for , and define to be the same for .

Theorem 3.2.

Let be a sequence such that and with probability going to 1 as . Assume such that as and almost surely. If , then, for any distribution of . Furthermore, if

Derivation of Theorems 1 and 2 may be found in Appendix C.

4 Simulation Studies

To illustrate the benefits of using crossover GOWL, we present simulation studies with comparisons to standard methods used in parallel group clinical trials. Simulated data sets were generated as follows. The covariates, , are i.i.d. variables drawn from a distribution. Subjects were randomized to treatment or for the parallel design or to sequence or for the crossover design with equal probability. The response for the parallel design,

is normally distributed with a mean of

and a variance of 1. For the crossover design, responses were simulated per the model , for , where was drawn from a multivariate normal distribution with mean , and . was fixed to be for all simulation scenarios. Table 1 describes choices of and defining four scenarios.

1 0 0
2 0 0
Table 1: The interactive and carryover effects for the five simulation scenarios.

Scenarios 1 and 3 are linear in , whereas Scenarios 2 and 4 are nonlinear. Note that scenario pairs and are similar, but Scenarios 3 and 4 include carryover effects. The optimal ITR was estimated via crossover GOWL, using a Gaussian kernel. The penalty parameter, and the Gaussian kernel bandwidth parameter, were selected using 5-fold cross-validation on the grids and , respectively. In scenarios where carryover effects are present, RLT (Zhu et al., 2015) was used to fit both models to estimate using Algorithm 1.

A testing data set of size was generated similarly with period 1 data only. The misclassification rate, or , of the estimated ITR applied to the testing set was calculated, where is the empirical mean in the test set. We also calculated the estimated value of the estimated ITR, (Qian and Murphy, 2011), where


Note that is constant here. The estimated value is the average reward observed under the estimated optimal ITR when applied to the testing set. Figure 3 in Appendix A displays the mean square error from estimating the carryover effects with RLT for Scenarios 3 and 4.

Simulations were repeated 1,000 times at training set sample sizes of 30, 75, 150, 300, and 600. Comparisons to various methods in the parallel setting at the same sample size are presented in Figures 1 and 2. These methods include OWL, GOWL, and ridge regression. For OWL and GOWL, a Gaussian kernel was used, and the aforementioned grids for

and are considered in 5-fold cross-validation. For ridge regression, the model includes all covariates and treatment-covariate interactions without any higher order terms or between-covariate interactions. 5-fold cross-validation was used to determine a value for the the ridge penalty parameter, where the same values for in the OWL methods are considered. All simulations were performed with R version 3.4.3 (R Core Team, 2017). RLT was implemented with the RLT package, version 3.2.1 (Zhu, 2017), and all OWL methods were implemented with the DynTxRegime package, version 3.2 (Holloway et al., 2018). While the DynTxRegime package does not currently support GOWL, the inputs for OWL can be recoded to implement GOWL. Ridge regression was carried out with the glmnet package (Friedman et al., 2010).

Figure 1 displays the average misclassification rates across all sample sizes, methods, and scenarios. Figure 2 displays the mean square error of the estimated value from the true value, i.e., from period 1 data. On average, crossover GOWL yields lower misclassification rates and higher estimated values at smaller sample sizes across all scenarios. Crossover GOWL shows marked improvement in both misclassification and estimated value for small . When is large, ridge regression yields competitive results with that from crossover GOWL, but crossover GOWL still appears to have marginal gains. Although GOWL in the parallel setting does not perform as well as OWL in any of the presented scenarios, Chen et al. (2018) discuss scenarios where improvements in misclassification and estimated value are observed when using GOWL as opposed to OWL.

Figure 1: Mean missclassification rate of 1,000 simulations for estimating the optimal ITR, applied to a testing set of size 10,000 for each of 4 simulation scenarios.
Figure 2: Mean square error of the estimated value compared to the true value from 1,000 simulations for estimating the optimal ITR, applied to a testing set of size 10,000 for each of 4 simulation scenarios.

5 FAME Feeding Trial Data Analysis

We present the application of crossover GOWL to data from the Food, Adolescents, Mood and Exercise (FAME) crossover feeding trial, conducted at the University of Southern California (USC) (O’Reilly et al., 2015). The FAME trial included African American and Latino adolescents who were overweight or obese. African American and Latino adolescents are disproportionately affected by overweight and obesity outcomes compared to their non-Hispanic counterparts (Ogden et al., 2014; O’Reilly et al., 2015; Taveras et al., 2013). Dietary intake is a major modifiable risk factor and represents a key intervention point in improving weight loss (Bleich et al., 2017; Kipping et al., 2008). One promising approach is to modify dietary components to improve satiety to indirectly reduce caloric intake (Anderson et al., 2009). In epidemiologic studies of adults in the US, fiber intake is inversely associated with body weight and body fat (Slavin, 2005), even after adjusting for confounding factors such as dietary fat intake. However, results from intervention studies are mixed: increased dietary fiber intake has been shown to have varied effects on body weight among adults who are overweight or obese, with limited research in pediatric or adolescent populations (Rössner et al., 1987; Ryttig et al., 1989; Slavin, 2005; Thompson et al., 2005; Tucker and Thomas, 2009). Given the heterogeneity in the effects of dietary fiber intake on body weight, it is essential to identify the subgroups of overweight and obese adolescents who may benefit from tailored clinical advice to increase fiber intake. We estimate a decision rule to identify a subgroup of adolescents who are overweight or obese that experiences larger increases in patient-reported satiety from a high fiber diet as opposed to the more common high sugar diet.

This study was conducted at the USC Health Sciences campus in Los Angeles, California from 2008 to 2011. Eighty-six Latino and African American adolescents (ages 14 to 17 years of age) who were overweight or obese (BMI percentile ) were recruited. Race was self-reported, and subjects were included if all four grandparents were Latino or African American. Subjects were excluded if they had type 2 diabetes, were in a weight loss program within the past 6 months, or used medications that influenced insulin or body composition. Informed written parental consent and participant assent were acquired before all testing procedures. The Institutional Review Board of USC approved all study procedures.

Participants received either a high sugar/low fiber (HSLF) meal plan or a high fiber/low sugar (HFLS) meal plan for breakfast and lunch on two separate visit days. Participants were randomized with equal probability to receive the HSLF/HFLS or HFLS/HSLF sequence with a minimum 2 week washout period between visits. The meals were isocaloric and matched for macronutrients except sugar and fiber content. Participants initially attended a baseline visit at the Clinical Trials Unit at the USC University Hospital where insulin sensitivity, Tanner stage via examination by a medical professional, BMI percentile for age, sex, ethnicity, waist circumference, and hemoglobin A1c (HbA1c) were collected. Insulin sensitivity was assessed via a frequently sampled intravenous glucose tolerance test (FSIVGTT) and calculated using the minimal model (Bergman et al., 1979; Yang et al., 1987). At the subsequent test meal visits, participants received either a HSLF or HFLS breakfast after a 10 hour overnight fast. At noon, the participants received the same meal condition for lunch. Participants rated their hunger and fullness via a 100 mm-visual analog scale (VAS) prior to breakfast and 45 minutes after the start of lunch (300 minutes after breakfast). Participants were provided with age appropriate activities between meals (e.g., video games, crafts, books, etc.).

The satiety outcomes are formally defined as the negative change in hunger, since lower values of hunger are desired, and the observed change in fullness between 8:00 AM and 1:00 PM (before breakfast and after lunch). Due to the nature of the outcomes, the required 10 hour overnight fast, and the implemented minimum 2 week washout period, we assumed no carryover effects were present. Of the 86 subjects who completed the study, 20 were removed for missing outcomes, and 1 was removed for missing insulin sensitivity. Participants that did not return within 5 weeks were also removed . We compared crossover GOWL with OWL, GOWL, and ridge regression using data from period 1 only. Methods were implemented as described in Section 4. 5-fold cross-validated value estimates were obtained, but rather than using Equation (6) which uses only period 1 data, the value for each observation in the th fold’s testing set was computed as where is the size of the th fold for Although OWL, GOWL, and ridge regression were trained on period 1 data, data from both periods were used to improve accuracy in the value estimate because the testing set size for each fold is quite small.

Resulting estimated values, averaged across folds, are presented in Table 2

along with the mean outcome observed from period 1. For both outcomes, all methods show improvement in the estimated value in comparison to randomization, but crossover GOWL yields the highest improvement. For self-reported fullness, crossover GOWL also yields the smallest standard deviation. When training crossover GOWL on the full dataset, 92% (51%) of participants are assigned to the HFLS to maximize the change in fullness (hunger). The distribution of features across the groups assigned to HFLS and HSLF from crossover GOWL for both outcomes are presented in Figure 4 in Appendix B. Dietary fiber is recommended to improve overall health in the general population

(Marlett et al., 2002); however, the estimated ITRs from hunger and fullness may inform the development of tailored dietary intake advice for subgroups of at-risk adolescents.

Fullness Hunger
Crossover GOWL
Table 2: Mean (sd) 5-fold cross-validated estimated values for feeding trial data compared with the observed value from period 1.

6 Discussion

Precision medicine is an emerging field with rapid developments in analytical methods; however, these advancements typically revolve around parallel designs. This paper proposes the combined use of crossover designs and generalized outcome weighted learning for the purpose of estimating optimal ITRs. The proposed method addresses a key gap in the literature; little to no work has been done to better involve crossover designs in precision medicine, despite how naturally crossover studies lend themselves to the field. Kulasekera and Siriwardhana (2018) propose a ranking method to estimate the optimal ITR from a crossover study but provide no recommendations on how to deal with carryover effects. In contrast, crossover GOWL is able to handle such effects. Furthermore, regardless of the presence of carryover effects, the proposed method shows improvements in the estimated value and misclassification rate, especially at the smaller sample sizes typical of crossover designs compared to standard methods with the parallel group design.

An alternative to GOWL that has been developed is residual weighted learning (RWL) (Zhou et al., 2017). RWL is an extension of OWL that weights the misclassification error by residuals from a model fit to the outcome instead of the observed rewards themselves. Unlike GOWL, RWL uses a non-convex loss function that does not guarantee global minimization (Tao et al., 2005). In the proposed method, there is no need to include residuals in the weight, because the residuals would cancel when taking the difference between responses to each treatment. Thus, the proposed method avoids specifying a model for the main effect of the covariates.

We note that when the distribution of is poorly allocated, the cross-validation mechanism for estimating and may fail. If there is prior knowledge on the distribution of , investigators could adjust randomization probabilities when assigning patients to treatment sequences accordingly. Otherwise, it is possible for a training set to not observe at least one or . Lastly, there may be low power in testing at smaller sample sizes.

Several extensions of estimating the optimal ITR from crossover data are yet to be explored. For example, only the

design was studied in this paper. For larger design schemes, the proposed method could be implemented in a series of binary classifiers as in

Dietterich and Bakiri (1994). Alternatively, one could expand crossover GOWL to multi-category classification. There have been several developments in multi-category SVM (Lee et al., 2004; Zhu et al., 2004). More recently, Liang et al. (2018)

propose an outcome weighted deep learning method to estimate the optimal ITR for multiple treatments. Another possible extension is to consider the residual from modeling the treatment response difference as the observed reward.

Fu et al. (2016) and Zhou et al. (2017) have seen favorable results using residual weights, but further improvements may come from using the residuals in outcome weighted learning with the piece-wise hinge loss from GOWL. Finally, the proposed method could be improved upon with methods for variable selection. For example, the penalty could be imposed during optimization to simultaneously restrict model complexity and perform variable selection as suggested by Chen et al. (2018), Song et al. (2015), Xu et al. (2015), and Zhou et al. (2017).

7 Acknowledgments

The authors were supported in part by NCI P01 CA142538 and NCMHD P60 MD002254-01.


  • Anderson et al. (2009) Anderson, J. W., P. Baird, R. H. Davis, S. Ferreri, M. Knudtson, A. Koraym, V. Waters, and C. L. Williams (2009). Health benefits of dietary fiber. Nutrition Reviews 67(4), 188–205.
  • Anderson et al. (1994) Anderson, J. W., B. M. Smith, and N. J. Gustafson (1994). Health benefits and practical aspects of high-fiber diets. The American Journal of Clinical Nutrition 59(5), 1242S–1247S.
  • Ashley (2015) Ashley, E. A. (2015). The precision medicine initiative: a new national effort. Journal of the American Medical Association 313(21), 2119–2120.
  • Athey and Wager (2017) Athey, S. and S. Wager (2017). Efficient policy learning.
  • Bartlett et al. (2006) Bartlett, P. L., M. I. Jordan, and J. D. McAuliffe (2006). Convexity, classification, and risk bounds. Journal of the American Statistical Association 101(473), 138–156.
  • Bergman et al. (1979) Bergman, R. N., Y. Z. Ider, C. R. Bowden, and C. Cobelli (1979). Quantitative estimation of insulin sensitivity. American Journal of Physiology-Endocrinology And Metabolism 236(6), E667.
  • Bleich et al. (2017) Bleich, S. N., K. A. Vercammen, L. Y. Zatz, J. M. Frelier, C. B. Ebbeling, and A. Peeters (2017). Interventions to prevent global childhood overweight and obesity: A systematic review. The Lancet Diabetes & Endocrinology 6(4).
  • Chakraborty and Murphy (2014) Chakraborty, B. and S. A. Murphy (2014). Dynamic treatment regimes. Annual Review of Statistics and its Application 1, 447–464.
  • Chen et al. (2018) Chen, J., H. Fu, X. He, M. R. Kosorok, and Y. Liu (2018). Estimating individualized treatment rules for ordinal treatments. Biometrics 74(3).
  • Collins and Varmus (2015) Collins, F. S. and H. Varmus (2015). A new initiative on precision medicine. New England Journal of Medicine 372(9), 793–795.
  • Cortes and Vapnik (1995) Cortes, C. and V. Vapnik (1995). Support vector networks. Machine learning 20(3), 273–297.
  • Dietterich and Bakiri (1994) Dietterich, T. G. and G. Bakiri (1994). Solving multiclass learning problems via error-correcting output codes.

    Journal of Artificial Intelligence Research

     2, 263–286.
  • Fleiss (1989) Fleiss, J. L. (1989). A critique of recent research on the two-treatment crossover design. Controlled Clinical Trials 10(3), 237–243.
  • Friedman et al. (2010) Friedman, J., T. Hastie, and R. Tibshirani (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software 33(1), 1–22.
  • Fu et al. (2016) Fu, H., J. Zhou, and D. E. Faries (2016). Estimating optimal treatment regimes via subgroup identification in randomized control trials and observational studies. Statistics in Medicine 35(19), 3285–3302.
  • Halliday et al. (2018) Halliday, T. M., S. V. Liu, L. B. Moore, V. E. Hedrick, and B. M. Davy (2018). Adolescents perceive a low added sugar adequate fiber diet to be more satiating and equally palatable compared to a high added sugar low fiber diet in a randomized-crossover design controlled feeding pilot trial. Eating Behaviors 30, 9–15.
  • Holloway et al. (2018) Holloway, S. T., E. B. Laber, K. A. Linn, B. Zhang, M. Davidian, and A. A. Tsiatis (2018). DynTxRegime: Methods for Estimating Optimal Dynamic Treatment Regimes. R package version 3.2.
  • Kallus (2018) Kallus, N. (2018). Balanced policy evaluation and learning. In Advances in Neural Information Processing Systems, pp. 8908–8919.
  • Kimeldorf and Wahba (1970) Kimeldorf, G. S. and G. Wahba (1970). A correspondence between bayesian estimation on stochastic processes and smoothing by splines. The Annals of Mathematical Statistics 41(2), 495–502.
  • Kipping et al. (2008) Kipping, R. R., R. Jago, and D. A. Lawlor (2008). Obesity in children. part 1: Epidemiology, measurement, risk factors, and screening. BMJ 337, a1824.
  • Kosorok and Moodie (2015) Kosorok, M. R. and E. E. Moodie (2015). Adaptive Treatment Strategies in Practice: Planning Trials and Analyzing Data for Personalized Medicine, Volume 21. SIAM.
  • Kulasekera and Siriwardhana (2018) Kulasekera, K. B. and C. Siriwardhana (2018). Multi-response based personalized treatment selection with data from crossover designs for multiple treatments.
  • Laber and Zhao (2015) Laber, E. and Y. Zhao (2015). Tree-based methods for individualized treatment regimes. Biometrika 102(3), 501–514.
  • Laber et al. (2014) Laber, E. B., D. J. Lizotte, M. Qian, W. E. Pelham, and S. A. Murphy (2014). Dynamic treatment regimes: Technical challenges and applications. Electronic Journal of Statistics 8(1), 1225.
  • Lavori and Dawson (2014) Lavori, P. W. and R. Dawson (2014). Introduction to dynamic treatment strategies and sequential multiple assignment randomization. Clinical Trials 11(4), 393–399.
  • Lee et al. (2004) Lee, Y., Y. Lin, and G. Wahba (2004). Multicategory support vector machines: Theory and application to the classification of microarray data and satellite radiance data. Journal of the American Statistical Association 99(465), 67–81.
  • Liang et al. (2018) Liang, M., T. Ye, and H. Fu (2018). Estimating individualized optimal combination therapies through outcome weighted deep learning algorithms. Statistics in Medicine 37(27), 3869–3886.
  • Lin (2002) Lin, Y. (2002). Support vector machines and the bayes rule in classification. Data Mining and Knowledge Discovery 6(3), 259–275.
  • Machin and Fayers (2010) Machin, D. and P. M. Fayers (2010). Randomized clinical trials: design, practice and reporting. John Wiley & Sons.
  • Marlett et al. (2002) Marlett, J. A., M. I. McBurney, and J. L. Slavin (2002). Position of the american dietetic association: health implications of dietary fiber. Journal of the American Dietetic Association 102(7), 993–1000.
  • Moodie et al. (2007) Moodie, E. E., T. S. Richardson, and D. A. Stephens (2007). Demystifying optimal dynamic treatment regimes. Biometrics 63(2), 447–455.
  • Ogden et al. (2014) Ogden, C. L., M. D. Carroll, B. K. Kit, and K. M. Flegal (2014). Prevalence of childhood and adult obesity in the united states, 2011-2012. Journal of the American Medical Association 311(8), 806–814.
  • O’Reilly et al. (2015) O’Reilly, G. A., B. R. Belcher, J. N. Davis, L. T. Martinez, J. Huh, L. Antunez-Castillo, M. Weigensberg, M. I. Goran, and D. Spruijt-Metz (2015). Effects of high-sugar and high-fiber meals on physical activity behaviors in latino and african american adolescents. Obesity 23(9), 1886–1894.
  • Petersen et al. (2007) Petersen, M. L., S. G. Deeks, and M. J. van der Laan (2007). Individualized treatment rules: Generating candidate clinical trials. Statistics in Medicine 26(25), 4578–4601.
  • Qian and Murphy (2011) Qian, M. and S. A. Murphy (2011). Performance guarantees for individualized treatment rules. Annals of Statistics 39(2), 1180.
  • R Core Team (2017) R Core Team (2017). R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing.
  • Rössner et al. (1987) Rössner, S., D. Von Zweigbergk, A. Öhlin, and K. Ryttig (1987). Weight reduction with dietary fibre supplements: Results of two double-blind randomized studies. Acta Medica Scandinavica 222(1), 83–88.
  • Rubin (1978) Rubin, D. B. (1978). Bayesian inference for causal effects: The role of randomization. The Annals of Statistics 6(1), 34–58.
  • Ryttig et al. (1989) Ryttig, K., G. Tellnes, L. Haegh, E. Bøe, and H. Fagerthun (1989). A dietary fibre supplement and weight maintenance after weight reduction: A randomized, double-blind, placebo-controlled long-term trial. International Journal of Obesity 13(2), 165–171.
  • Slavin (2005) Slavin, J. L. (2005). Dietary fiber and body weight. Nutrition 21(3), 411–418.
  • Song et al. (2015) Song, R., M. Kosorok, D. Zeng, Y. Zhao, E. Laber, and M. Yuan (2015). On sparse representation for optimal individualized treatment selection with penalized outcome weighted learning. Stat 4(1), 59–68.
  • Tao et al. (2005) Tao, P. D. et al. (2005). The dc (difference of convex functions) programming and dca revisited with dc models of real world nonconvex optimization problems. Annals of Operations Research 133(1-4), 23–46.
  • Taveras et al. (2013) Taveras, E. M., M. W. Gillman, K. P. Kleinman, J. W. Rich-Edwards, and S. L. Rifas-Shiman (2013). Reducing racial/ethnic disparities in childhood obesity: The role of early life risk factors. Journal of the American Medical Association Pediatrics 167(8), 731–738.
  • Thompson et al. (2005) Thompson, W. G., N. R. Holdman, D. J. Janzow, J. M. Slezak, K. L. Morris, and M. B. Zemel (2005). Effect of energy-reduced diets high in dairy products and fiber on weight loss in obese adults. Obesity Research 13(8), 1344–1353.
  • Tucker and Thomas (2009) Tucker, L. A. and K. S. Thomas (2009). Increasing total fiber intake reduces risk of weight and fat gains in women. The Journal of Nutrition 139(3), 576–581.
  • Turner (2010) Turner, J. R. (2010). New drug development: An introduction to clinical trials. Springer Science & Business Media.
  • US Department of Agriculture (2010) US Department of Agriculture (2010). Us department of health and human services. dietary guidelines for americans.
  • Wellek and Blettner (2012) Wellek, S. and M. Blettner (2012). On the proper use of the crossover design in clinical trials: Part 18 of a series on evaluation of scientific publications. Deutsches Ärzteblatt International 109(15), 276.
  • Xu et al. (2015) Xu, Y., M. Yu, Y.-Q. Zhao, Q. Li, S. Wang, and J. Shao (2015). Regularized outcome weighted subgroup identification for differential treatment effects. Biometrics 71(3), 645–653.
  • Yang et al. (1987) Yang, Y. J., J. H. Youn, and R. N. Bergman (1987). Modified protocols improve insulin sensitivity estimation using the minimal model. American Journal of Physiology-Endocrinology And Metabolism 253(6), E595–E602.
  • Zhang et al. (012b) Zhang, B., A. A. Tsiatis, M. Davidian, M. Zhang, and E. Laber (2012b). Estimating optimal treatment regimes from a classification perspective. Stat 1(1), 103–114.
  • Zhang et al. (012a) Zhang, B., A. A. Tsiatis, E. B. Laber, and M. Davidian (2012a). A robust method for estimating optimal treatment regimes. Biometrics 68(4), 1010–1018.
  • Zhao and Zeng (2013) Zhao, Y. and D. Zeng (2013). Recent development on statistical methods for personalized medicine discovery. Frontiers of Medicine 7(1), 102–110.
  • Zhao et al. (2012) Zhao, Y., D. Zeng, A. J. Rush, and M. R. Kosorok (2012). Estimating individualized treatment rules using outcome weighted learning. Journal of the American Statistical Association 107(499), 1106–1118.
  • Zhou et al. (2017) Zhou, X., N. Mayer-Hamblett, U. Khan, and M. R. Kosorok (2017). Residual weighted learning for estimating individualized treatment rules. Journal of the American Statistical Association 112(517), 169–187.
  • Zhu et al. (2004) Zhu, J., S. Rosset, R. Tibshirani, and T. J. Hastie (2004). 1-norm support vector machines. In Advances in Neural Information Processing Systems, pp. 49–56.
  • Zhu (2017) Zhu, R. (2017). Reinforcement Learning Trees. R package version 3.2.1.
  • Zhu et al. (2015) Zhu, R., D. Zeng, and M. R. Kosorok (2015). Reinforcement learning trees. Journal of the American Statistical Association 110(512), 1770–1784.


Appendix A

The reinforcement learning trees (RLT) (Zhu et al., 2015) performance from simulation Scenarios 3 and 4 as visualized in Figure 3, which displays the mean square prediction error of the estimated carryover compared with the true carryover from the testing set, or

Despite the potential for a high mean square error, the crossover design still outperforms parallel design counterparts despite the presence of carryover effects, as can be seen in Figures 1 and 2.

Figure 3: Boxplots of mean square error for RLT predicted on the testing set compared with the true carryover.

Appendix B

92% of study participants, were assigned to the HFLS diet according to crossover GOWL to maximize change in fullness from baseline. To characterize the subgroup that, on average, experiences a larger increase in patient reported fullness, Figure 4 displays the distribution of continuous features across the estimated subgroups. Those assigned to the HFLS diet tend to be older with higher A1c. Because the HSLF group is small (), two-sample -tests would not be appropriate to test for significant differences between groups, and trends observed in Figure 4 should be confirmed in future studies. However, sex , ethnicity , and Tanner stage were tested using Fisher’s exact tests. All tests were non significant at the 0.05 level.

Figure 4: Distribution of features across the crossover GOWL estimated diet-outcome subgroups for both fullness and hunger outcomes

Figure 4 also displays the distribution of continuous features across the estimated subgroups to minimize the change in hunger from baseline. 51% of participants were assigned to HFLS. Those assigned to HFLS tend to be older, but differences in other covariates are not apparent. Although the sample in the HSLF is larger when we consider hunger as the outcome, both samples are still rather small. For this reason, two-sample -tests are still not appropriate. Fisher’s exact tests again did not yield any significant differences in sex , ethnicity , or Tanner stage .

In conclusion, using crossover GOWL appears to be effective for estimating the optimal ITR to maximize the change in satiety. Future research should confirm these subgroups in large sample sizes to better compare differences across features. If verified, future recommendations for adolescent minorities can be tailored by age and A1c levels to improve weight loss. Studies on overweight and obese minority adolescents are still needed to research alternative interventions for those that report feeling more satiated from the typical Western diet (HSLF).

Appendix C

The following assumptions are made for the theory behind the method proposed in the main paper.

  1. Positivity: with probability 1

  2. Conditional Exchangeability:

  3. Consistency:

  4. Outcomes follow the model

    for periods . has a positive definite covariance matrix,

  5. There exist such that almost surely, almost surely, and as

Proof of Lemma 1. The optimal ITR is . Note that, under Assumption (4), The expected treatment response difference between treating according to and treating opposite to is

for all . Thus, the optimal ITR also maximizes the treatment-response difference, or . Therefore, it can be seen that

where the second equality follows from Assumption (3). This proves the result.

Proof of Theorem 1. This proof follows from Lemma 1 and the results from Lin (2002). Recall that . Minimizing the risk, is equivalent to minimizing the conditional risk,

for every fixed . Let and By the law of total expectation, the conditional risk becomes

Next, note that whenever For example, when , the conditional risk reduces to

which is monotonically increasing as . A similar argument is made for when . Thus, we restrict our search to . Then,

If minimizes the conditional risk, then must have the sign opposite of the expression , and thus

Proof of Theorem 2. First, define the loss functions


Next, we show that is bounded. For any

by definition of If we choose , then, for all large enough,

where the last inequality holds because of Assumptions (1), (5), and (6). Define Then, because we have that

For any bounded , such as , we may show that

Next, we have

Taking the on both sides, we find

Thus, it suffices to show that . Because is bounded by , is contained in a Donsker class. Note that is Lipschitz continuous with respect to , and, thus, is Lipschitz continuous with respect to . Therefore, is also Donsker. This gives us , which implies We finally arrive at . Furthermore, when , and

where the first inequality holds from Bartlett et al. (2006).