Precision medicine (PM) consists in associating the most appropriate treatment to each patient according to his or her characteristics. This is, for instance, quite common in the clinical management of cancer patients where the choice of treatment is increasingly influenced by the genomic alterations of the patient . At the individual level, targeted treatments has provided relevant solutions for patients with specific mutations . Putting together these various treatments, some precision medicine strategies can be defined: based on the omics profile of the patient, the treatment most likely to be successful is chosen. If the information available is reliable, precision medicine can thus be reduced to a treatment choice algorithm that takes as input the molecular characteristics of the patient’s tumour and outputs a recommendation of treatment.
The question then arises of how to quantify the clinical benefit provided by these treatment algorithms. Some clinical trials have been proposed, demonstrating both the feasibility of collecting information about mutations  or RNA  in real-time and the clinical benefit that can be expected from these approaches for some patients . However, the increasing abundance of omics data and biological knowledge make it progressively easier to establish new algorithms for precision medicine, either directly based on physician knowledge or provided by computational models . For practical reasons it is not possible to propose a real clinical trial for each new precision medicine algorithm or for any variants, comparing standard of care with new algorithm-based treatments.
Therefore, this work provides a method to assess the clinical impact of proposed PM treatment algorithm based on already generated data, emulating clinical trials and analyzing them in the causal inference framework . First we will define the causal estimates of the precision medicine effects (later referred to as causal estimates) we want to assess, and the corresponding ideal clinical trials one would like to perform. Next, we will define the notations and the causal framework we use to infer the causal effects from observational data with multiple versions of treatment, based on the previous work by . It will be briefly introduced in its main principles and then extended in order to adapt to the characteristics of PM, in particular the multiplicity of treatment versions, i.e. targeted drugs. Then we will apply the proposed methods to simulated data in order to investigate the different biases of the candidate methods. An example scenario will be presented and a RShiny interactive application has been developed to further explore other user-defined settings. Finally, the analysis of data from patient-derived xenografts (PDX) makes it possible both to apply the methods to pre-clinical situation and to have data approximating the counterfactual responses, thus enabling further validation of the proposed estimation methods.
2 Target trials for precision medicine: definition of causal estimates
We first specify the precision medicine effects that are to be estimated. These effects will finally be estimated based on observational data through the causal framework and target trial emulation . Thus, if we think in terms of clinical trials, we are not trying to prove or quantify the superiority of one treatment over another but rather to evaluate the clinical utility of a precision medicine strategy assigning treatments based on genomic features of patients. This is therefore closer to the well-studied biomarker-based designs for clinical trials . In a way, it is a matter of extending these unidimensional biomarker-based designs to multidimensional strategies that allow a choice between quite a number of different treatments. The potentially large number of treatments thus prompts us to draw more inspiration from scalable biomarker-strategy designs than biomarker-stratified designs . We can draw a methodological parallel with some trials like the Tumor Chemosensitivity Assay Ovarian Cancer study in which a biochemical assay guides the choice of preferred chemotherapy for patients in a panel of twelve different treatments . More recently, some clinical trials have been proposed that include precision medicine strategies, particularly in oncology [3, 11].
On the basis of these clinical examples, we propose three different target trials and their corresponding causal estimates, the clinical relevance of which may vary according to medical contexts. Each target trial contains a precision-medicine directed arm in which patients are treated in accordance with the precision medicine algorithm recommendations but they are differentiated from each other by alternative control arms (Figure 1). Causal effects will be estimated solely on patients eligible for the assignment of a personalized treatment, i.e. those for whom the treatment algorithm is able to recommend a drug.
2.1 First causal effect (): comparison with a single standard
The first possible target trial is to compare the precision medicine arm with a control arm in which all patients have been treated with the same single treatment. This could classically be the current standard of care applied to all patients (e.g chemotherapy cancer treatment).
2.2 Second causal effect (): comparison with physician’s assignment of drugs
Then, in order to propose a more comprehensive clinical assessment, we propose a second causal effect, comparing the PM arm with the current clinical practice, i.e the assignment of the same targeted treatments by physicians in the absence of the algorithm. This implicitly means comparing two PM strategies: the one derived from the algorithm and the one that corresponds to current physician’s knowledge. Unlike the former, the latter may not be perfectly deterministic depending on the heterogeneity of medical knowledge or practices. This way of defining by focusing on the doctor’s assignment of the same treatments stems from our question of interest: to quantify the relevance of the algorithm itself. Another vision would have been to compare the precision medicine arm with the doctor’s treatments, allowing him to use treatments other than those of the PM arm, such as the gold-standard one described in . But the differences between the arms could then be biased by the use of treatments with different overall efficacy, changing the focus of the question. We will therefore stick to the first definition, which is more focused on the relevance of the algorithm.
2.3 Third causal effect (): comparison with random assignment of drugs
Finally, we define the effect comparing the PM arm with a control arm using exactly the same pool of treatments assigned randomly. In this case, we measure the ability of the PM algorithm to assign treatments effectively based on genomic features of patients. This comparison has already been considered in the context of biomarker-based clinical trials . Although this comparison with random assignment is methodologically relevant, it may not make sense from a clinical point of view if the common clinical practice already contains strong indications (or contraindications) for some patient-treatment associations.
3 Precision medicine causal inference with multiple versions of treatment
3.1 Causal inference and the potential outcomes framework
Now that we have defined what we want to estimate, we need to specify the method of estimation. In cases where it would be too difficult or too early to conduct a true clinical trial we propose methods using observational data to emulate them. Indeed, it is possible to derive estimates with a causal interpretation from observational data in the context of the potential outcomes framework . We will first describe briefly the fundamentals of this framework before extending this framework to our precision medicine settings in section 3.2.
3.1.1 First notations and causal graph
We will use to index the individuals in the population. and correspond respectively to the actual treatment received by individual and the outcome. In the most simple case, treatment takes values in , denoting the treated patients and the control ones. corresponds to the patient’s response to treatment. In the case of cancer it may be a continuous value (e.g size of tumour), a binary value (e.g status or event indicator), or even a time-to-event (e.g time to relapse or death). Only the first two cases will be discussed later. Finally, it is necessary to take into account the possible presence of confounders influencing both and and denoted for individual . These simple relations can be summarized in a causal graph (Figure 2).
3.1.2 Potential outcomes framework
One standard framework to estimate causal effects relies on potential outcomes . This framework is sometimes described as counterfactual because it defines variables like to denote the potential outcome of individual in case he has been treated by which may be different from what we observe if . These counterfactual variables make it possible to write the causal estimands. For instance, in this context, we can easily compute the difference in outcome between treated patients and control patients:
However, this difference has no causal interpretation as it does not offer any guarantees as to the confounding factor, as an unbalanced distribution of can induce biases Thus we define another estimate:
In this case, we compare between two ideal cohort, one in which all patients have been treated (possibly contrary to the fact) and one in which all patients have been left in the control arm (once again, possibly contrary to the fact). Under certain assumptions of consistency, positivity and conditional exchangeability, the potential outcomes framework allows to estimate these counterfactual variables and therefore infer causal estimates from observational (non-randomized) data [13, 14].
Consistency means that values of treatment under comparison represent well-defined interventions which themselves correspond to the treatments in the data:
Exchangeability means that treated and control patients are exchangeable, i.e if the treated patients had not been treated they would have had the same outcomes as the controls, and conversely. Since we usually observe some confounders we define conditional exchangeability to hold if cohorts are exchangeable for same values of confounding . Therefore conditional exchangeability will hold if there are no unmeasured confounding:
assumption states that the probability of being administered a certain version of treatment conditional onis greater than zero: Intuitively, this positivity condition is required to ensure that the defined counterfactual variables make sense and do not represent something that cannot exist.
3.1.3 Identification of causal effects
Different methods provide estimators to evaluate causal effect from observational data. Throughout the article, we will describe essentially one method called standardization or parametric g-formula. Details on other types of estimators are available in Supplementary Materials, sections B and C. In this simple case (Figure 2), the causal effect of treatment can be written with standardized means (formal proof in Supplementary Materials, section A):
Computationally, non-parametric estimation of is usually out of reach. Thus, on real-world dataset, is estimated through modelling and explicit computation is replaced by its empirical estimate.
3.2 Precision medicine and the multiple versions of treatment
3.2.1 A treatment with multiple versions
The statement of the theoretical framework implicitly implies the uniqueness of the versions of the treatment  or at least the treatment variation irrelevance . In the precision medicine case, the multiplicity of versions is inherent: a given treatment status may encompass several drugs since a patient may be associated with several molecular agents based on his or her genomic characteristics. can be seen as a compound treatment  or a treatment with multiple versions .
Therefore, we define a variable denoting the version of treatment administered to individual . If is the arm to which the patient is assigned, is the molecule received, the version of treatment (e.g a specific anti-cancer drug) and , the set of versions of treatment . In our precision medicine problem, will denote control patients and the patients treated with an anti-cancer drug of the precision medicine pool. is the set of possible targeted treatments for patients. For the sake of simplicity we will assume that there is only one treatment version for controls, .
We also need to define other counterfactual variables like , the counterfactual version of treatment if the subject had been given the treatment level . Thus, we finally write the counterfactual outcome as for individual when treatment has been set to , using as the version of treatment , with . Causal relations between variables , , and are depicted in the causal diagram in Figure 3. It should be noted that has no direct influence on , its only effect is entirely mediated by , which is the real treatment in the pharmacological sense.
In this context, we can also define the assignment of a version of treatment for patients eligible to precision medicine algorithm. It is important to note that not all patients are necessarily eligible for the precision medicine strategy. Indeed, the treatment assignment algorithm relies on targetable alterations to establish its recommendations. In the absence of these, no recommendation can be offered to the patient. We denote the set of eligible patient profiles and consequently define the drug assignment algorithm as the function which associates to each a precision medicine treatment version such as:
3.2.2 Causal inference with multiple versions
Consequently, the multiplicity of versions prevents direct application of the framework as described in section 3.1. The theoretical framework has however been extended to causal inference under multiple versions of treatment and some identifiability conditions and properties have been studied, especially in the seminal article by . One of the first required adaptation to identify some causal effects is to partition confounders and (Figure 3). indicates a collection of covariates that may be causes of treatment or version of treatment but are not direct causes of . These covariates are of special interest for causal effects identification under multiple versions of treatment. indicates all other covariates. In our precision medicine settings, the genomic features of patients may define the eligibility to precision medicine and therefore affect . They may also be used to define the version of treatment . And finally they can influence the response to treatment . Thus, the genomics features of patients, as defined in section 1, are a typical example of type confounders. All causal relationships are summarized in Figure 3. Please note that all subsequent definitions are made taking into account W. However, no such variable is present in the application provided.
We summarize here some general observations from  regarding the extension of the framework to multiple versions before discussing specific estimates of interest of our precision medicine settings in section 3.2.3. First of all, the identifiability conditions have to be adapted. The consistency assumption for instance is extended to :
Then, the conditional exchangeability or no-unmeasured confounding assumptions, may be stated in two different ways, either without or with versions of treatment:
Specifically, it should be noted that we need to add in the set of covariates that must be taken into account in standardization, and we need positivity to hold for and , i.e. . Detailed proof of equation 4 is provided in Supplementary Materials, section A. Equation 4 paves the way to overall treatment effect assessment since would estimate the effect of treatment compared to with current versions of treatment.
Similarly, we can define
a random variable for versions of treatment with conditional distributionand assuming the equation 3 to be true we can derive the following formula and its formal proof in Supplementary Materials, section A:
In this case, to allow estimation of the right-hand side of the equation, positivity will be defined as .
3.2.3 Application to precision medicine
In the context of the potential outcomes framework extended to treatments with multiple versions, it is therefore possible to apply equations 4 and 6 in order to define and estimate the precision medicine causal effects previously described in section 2.
corresponds to control patients with and to patients treated with a targeted treatments. It is important to notice that from this point on we systematically restrict ourselves to patients eligible to the precision medicine algorithm, i.e. to individuals such as .
is a comparison between the precision medicine arm and a single version control arm:
In details, can be derived from equation 6 in the case where if and otherwise:
Then, and can be derived from equation 5:
Alternatively, if one wants to use as control only one of the treatments used in the PM arm the previous estimate could be replaced by the following one:
It should be noted that , like and presented later, depends on the PM algorithm of interest . could therefore also be written .
Then, is written using the PM targeted treatment that would have been assigned to the patient by the physician if the patient had been allocated in arm with PM targeted treatments:
is derived from equation 4:
Defining as the random distribution of versions of treatment , expresses as:
denoting the cardinality of the set. In this formula, can be derived from equation 6:
3.3 Alternative estimation methods
For the sake of simplicity and brevity, we detailed the standardization. However, other popular candidate methods can be used. Estimators based on the inverse probability weighting (IPW) and targeted maximum likelihood estimation (TMLE) will also be computed in the following sections. A description of the theoretical framework of these two approaches and their adaptation to multiple versions of treatment is provided in Supplementary Materials, sections B and C.
3.4 Code implementation
Computation of causal effects is implemented in R and the code is provided in the form of R notebooks (simulated data and PDX data) as well as in the form of an RShiny application (simulated data only). All of these files are available in the dedicated GitHub: https://github.com/JonasBeal/Causal_Precision_Medicine.
4 Simulation study
The proposed methods are first tested on simulated data in order to check the performance of the estimators in finite sample sizes.
4.1 General settings
|Response variable||Intercept||Linear regression coeff.|
Using the R package lava, we simulate a super-population of 10000 patients with variables C, A, K and Y as in 3
. We first define two independent binary variablesand , representing mutational status of genomic covariates, with a prevalence of 40%. By analogy with the PDX data presented in the next section, Y mimics the evolution of tumour volume and a low value (a fortiori negative) corresponds to a better response. Y is therefore defined as a continuous gaussian variable. For each counterfactual variable of response , we specify the intercept and the linear regression coefficients regarding influence of as described in Table 1. Lower intercepts correspond to better responses/more efficient drugs. Similarly, a negative regression coefficient between and means that the gene improves the response to . So all in all, has the best basal response (lowest intercept). (resp. ) improves the response to (resp. ). The treatment algorithm of precision medicine is in line with these settings since patients mutated in (regardless their status) are recommended to take and patients mutated for only are recommended to take . Patients without mutations are not eligible to precision medicine and not taken into account in the computations. Since has the bast basal response we assume it is assigned with greater probability by the physician and implement the following distribution of observed treatments:
A super-population of 10000 patients is then generated. 1000 cohorts of 200 patients are sampled without replacement within this super-population which, with the prevalences defined for the mutations, corresponds to an effective sample size of about 130 patients eligible to the PM algortithm. the causal effects , and are computed based on different methods on the sub-cohort eligible to precision medicine:
True effects, using counterfactuals for all patients
Naive effect, using observed outcomes only for both arms
Corrected effects: using observed outcome, computed with standardized estimators (Std), inverse probability weighting (IPW) and targeted maximum likelihood estimators (TMLE). Details about estimators of the last methods are available in Supplementary Materials, sections B and C.
4.2 Simulation results
First, the distribution of data in the super-population of 10,000 patients can be observed in Figure 4A, illustrating the different relations and differences described above. In particular, (resp. ) is lower for -mutated (resp. -mutated) patients. It can also be seen that the response to precision medicine () differs according to the groups: patients mutated for only have the best response, followed by patients mutated for both and and patients mutated for only. There is therefore a heterogeneity of responses to PM which encourages to take into account the groups of patients and their PM versions. The right side of Figure 4A shows the deterministic assignment of the recommended PM treatment () to each patient profile and the unbalanced distribution of observed treatments () with a predominance of .
In the first target trial, true estimates in the sampled cohorts are distributed around -40 (Figure 4B), confirming the superiority of the PM arm over the control arm as defined in the simulation parameters. Not all methods of estimating the causal effect perform equally well. The so-called naive estimate and the one based on IPW show a net bias. The over-representation of the most advantaged patients by PM tends to cause these methods to overestimate the benefit of PM, as can also be seen in the deviation plots. The same trends are observed for and (Figure 4C and D) where the differences are even more drastic. The mean absolute error of the naive method is thus divided by more than 2 when using standardized estimates or the TMLE.
In order to further dissect the influence of simulation parameters on estimation performances, a slightly different simulation scenario with equal probabilities of observed treatments is provided in Supplementary Materials, section D. This case shows that a random and balanced assignment of the observed treatments logically removes the systematic biases of the naive method by providing them with more randomized data. However, the corrections made by the proposed methods of causal inference, and in particular standardization and TMLE, reduce the variances in the estimates due to the heterogeneity of the effects of precision medicine as a function of molecular profiles.
5 Application to pre-clinical data: patient-derived xenografts (PDX)
The method is then applied to public data from patient-derived xenografts . Each patient tumour is divided into pieces later implanted in several immunodeficient cloned mice treated with different drugs, thus providing access to sensitivities to several different drugs for each tumour (see Supplementary Materials, section E). The original dataset contains 281 different tumours of origin (sometimes called PDX models, in the sense of a biological model) and 63 tested drugs, not all drugs having been tested for all tumours . 192 of these tumours have also been characterized for their mutations, copy-number alterations and mRNA.
Such data provides access to treatment response values otherwise considered as hypothetical (or counterfactual). Availability of these data provides a unique ground truth to assess the validity of proposed causal estimates in a pre-clinical context. Based on the analysis accompanying the published data , some biomarkers of treatment response have been selected and resulted in an example of a treatment algorithm: binimetinib (MEK inhibitor) is recommended to KRAS/BRAF mutated tumours, and BYL719 (alpha-specific PI3K inhibitor, also known as Alpelisib) to PIK3CA mutated tumours. PTEN is also included as a covariate because of its detrimental impact on the response to these two treatments. LEE011 drug (a cell cycle inhibitor also known as Ribociclib) is chosen as the reference drug treatment (). It should be noted that different drug response metrics are computed in the source data, two of which will be used in this case study. The first one is continuous and called BestAverageResponse
in the data, it is based on the variation of the tumour volume after treatment, the lower values (and especially negative) corresponding to better responses. The second one is originally categorical and based on a modified Response Evaluation Criteria In Solid Tumors (RECIST) criteria. It was binarized for this study so that the responders have a score of 1 and non-responders 0. The details of the definition and distribution of these metrics are given in Supplementary Materials, section E. Among the sequenced tumours, 88 are eligible to this precision medicine algorithm (i.e. mutated for BRAF, KRAS or PIK3CA) and have been tested for all 3 drugs of interest, thus ensuring a the availability of all responses of interest. The following analyses will focus exclusively on this sub-cohort for which a comprehensive analysis is provided in Supplementary Materials, section E.
The analysis settings are similar to the ones used for simulated data. 1000 different cohorts of 70 tumours (out of 88) are sampled without replacement assuming each time that only the response to one of the treatments is known for each tumour, reproducing the classical clinical situation. The distribution of the observed treatments was defined randomly:
It should be noted that, contrary to analyses based on simulated data, all the statistical models used for standardisation (outcome model), for the IPW (treatment model) and for the TMLE are no longer generalized linear models (GLM) but random forests. This is intended to avoid misspecification due to the largely non-linear nature of the data. It was also observed that the performance of GLM-based methods was indeed lower than that of the naive method, supporting the importance of relevant model specification consistent with real data.
The results of estimations are then presented in Figure 5. In the presence of randomly assigned and balanced observed treatments, none of the methods (including the naive one) has significant systematic bias. On the other hand, more sophisticated methods, and in particular TMLE, allow to reduce the gap between estimates and true values, which is achieved through the special design of the PDX data.
In synthesis, this work proposes a conceptual framework for evaluating a precision medicine algorithm, taking advantage of data already generated using adapted causal inference tools. In a clinical context, these data were not generated in a purely observational manner. Patients were cared for and treated by physicians who probably took into account some of their characteristics. However, the reasoning, formalized or not, behind the physicians’ decisions does not correspond to that which a new investigator might want to test. In the eyes of this new investigator, the data can therefore be considered as observational in that they do not correspond to the randomization he would have liked to have carried out. The possibility for this new investigator to estimate the impact of his PM algorithm using the proposed estimators depends, however, on the consistency, exchangeability and positivity hypotheses.
The hypothesis of consistency has been made more plausible by taking into account the treatment versions, which makes it possible to explicit the heterogeneity of the molecules administered. Exchangeability remains questionable. The simulations and calculations described above underline the importance of taking into account at least the genomic covariates used in the processing algorithm. The inclusion of additional covariates is likely to be necessary in many real-world applications. Positivity, on the other hand, can be violated in a much more obvious way in certain situations. Thus, equation 6 requires positivity to be extended to versions of treatment: . If the assignment of the observed treatments was done on a deterministic basis with respect to the variables used by the treatment algorithm, each patient’s molecular profile will have been treated with a single drug, thus preventing any subsequent causal inference within the defined framework. The eventual use, by the boards of physicians in charge of assigning the observed treatments, of variables different from those used by the algorithm could then make it possible to verify the positive condition. But these variables would represent unmeasured confounding factors. It is therefore essential to have an in-depth knowledge of the rationales at work in the assignment of the observed treatments.
We developed a user-friendly application that extends the scope of the simulations and makes possible to study and quantify the impact of different situations, including possible (quasi-) violations of positivity or unmeasured confounding. It is thus a tool for empirically framing cases where this causal inference is reasonable or not. The analysis of the PDX data provides an illustration and proof of feasibility for these methods on pre-clinical data, closer to the human clinical data generally of interest. Beyond feasibility, this implementation leads to some remarks. Firstly, the improvement of causal inference methods compared to naive estimation of PM effects is conditioned in this case to the use of flexible and non-linear learning algorithms. This underlines the importance of a proper specification of the outcome and treatment models whose imperfection, especially when trained on small samples, could explain the modesty of the results compared to the simulated data. The particular nature of the PDX data design used should also be kept in mind: each tumour is tested only once for each drug, which may lead to greater variability of results due to tumour heterogeneity . Some studies, with smaller numbers of tumours and treatments, propose to form groups of several mice for each treatment-drug combination . The use of these mean effects could contribute to more accurate data. In spite of these limitations, which may diminish their ability to provide values with counterfactual interpretation, PDX data are thus a dataset of interest for studying and validating methods of causal inferences about treatment response. It can also be noted that the very nature of these data, due to the multiplicity of drugs tested for each tumour, can provide a framework in which the constraints of positivity are singularly alleviated. Even if all drugs were not tested on all patients, considering each tumour-drug combination as a different unit increases the coverage of the data. It is then necessary to take into account the clustered nature of the data, each tumour being present several times.
Finally, beyond the preclinical data presented here, the theoretical framework developed in this article should be more directly applicable to data from clinical trials if these data do not violate the requirements of positivity. If it is necessary to consider several trials, the heterogeneity of practices must be taken into account. The use of different drug lists from one trial to another or from one medical centre to another could also provide an example of confounding factor , included in the theoretical framework presented here but not used in applications.
All analyses have been performed with R. Analysis on simulated data is available as R code organized in a notebook and also as a RShiny interactive application designed to test different simulation scenarios. Analysis on PDX data is available as R notebook and the original dataset is also provided. All files and documentation are available on the dedicated GitHub repository: https://github.com/JonasBeal/Causal_Precision_Medicine. For the interactive application, readers unfamiliar with R can refer to the online version of the RShiny application: https://jonasbeal.shinyapps.io/application_causal_pm/.
8 Supplementary Materials
Theoretical details are given about standardised estimators in section A, about IPW estimators in section B and about TMLE in section C. Section D showcases an additional scenario with simulated data. An extensive analysis of PDX data is also provided in section E. Section F is focused on an example with a binary outcome.
The authors thank Christophe Le Tourneau, Stephen Cole and Margarita Moreno-Betancur for helpful comments and discussions.
Conflict of Interest: None declared.
-  Adam A Friedman, Anthony Letai, David E Fisher, and Keith T Flaherty. Precision medicine for cancer with next-generation functional diagnostics. Nature Reviews Cancer, 15(12):747–756, 2015.
-  Rony Abou-Jawde, Toni Choueiri, Carlos Alemany, and Tarek Mekhail. An overview of targeted treatments in cancer. Clinical therapeutics, 25(8):2121–2137, 2003.
-  Christophe Le Tourneau, Jean-Pierre Delord, Anthony Gonçalves, Céline Gavoille, Coraline Dubot, Nicolas Isambert, Mario Campone, Olivier Trédan, Marie-Ange Massiani, Cécile Mauborgne, et al. Molecularly targeted therapy based on tumour molecular profiling versus conventional therapy for advanced cancer (shiva): a multicentre, open-label, proof-of-concept, randomised, controlled phase 2 trial. The lancet oncology, 16(13):1324–1334, 2015.
-  Jordi Rodon, Jean-Charles Soria, Raanan Berger, Wilson H Miller, Eitan Rubin, Aleksandra Kugel, Apostolia Tsimberidou, Pierre Saintigny, Aliza Ackerstein, Irene Braña, et al. Genomic and transcriptomic profiling expands precision cancer medicine: the winther trial. Nature medicine, 25(5):751, 2019.
-  Geraldine O?Sullivan Coyne, Naoko Takebe, and Alice P Chen. Defining precision: the precision medicine initiative trials nci-mpact and nci-match. Current problems in cancer, 41(3):182–193, 2017.
-  J Hansen and R Iyengar. Computation as the mechanistic bridge between precision medicine and systems therapeutics. Clinical Pharmacology & Therapeutics, 93(1):117–128, 2013.
-  Miguel A Hernán and James M Robins. Using big data to emulate a target trial when a randomized trial is not available. American journal of epidemiology, 183(8):758–764, 2016.
-  Tyler J VanderWeele and Miguel A Hernan. Causal inference under multiple versions of treatment. Journal of causal inference, 1(1):1–20, 2013.
-  Boris Freidlin, Lisa M McShane, and Edward L Korn. Randomized clinical trials with biomarkers: design issues. Journal of the National Cancer Institute, 102(3):152–160, 2010.
-  Ian A Cree, Christian M Kurbacher, Alan Lamont, Andrew C Hindley, Sharon Love, TCA Ovarian Cancer Trial Group, et al. A prospective randomized controlled trial of tumour chemosensitivity assay directed chemotherapy versus physician’s choice in patients with recurrent platinum-resistant ovarian cancer. Anti-cancer drugs, 18(9):1093–1101, 2007.
-  Keith T Flaherty, Robert Gray, Alice Chen, Shuli Li, David Patton, Stanley R Hamilton, Paul M Williams, Edith P Mitchell, A John Iafrate, Jeffrey Sklar, et al. The molecular analysis for therapy choice (nci-match) trial: Lessons for genomic trial design. JNCI: Journal of the National Cancer Institute, 2020.
-  Daniel J Sargent, Barbara A Conley, Carmen Allegra, and Laurence Collette. Clinical trial designs for predictive marker validation in cancer treatment trials. Journal of Clinical Oncology, 23(9):2020–2027, 2005.
-  Donald B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 66(5):688, 1974.
-  MA Hernán and JM Robins. Causal inference: What if. Boca Raton: Chapman & Hill/CRC, 2020, 2020.
-  Donald B Rubin. Randomization analysis of experimental data: The fisher randomization test comment. Journal of the American Statistical Association, 75(371):591–593, 1980.
-  Tyler J VanderWeele. Concerning the consistency assumption in causal inference. Epidemiology, 20(6):880–883, 2009.
-  Miguel A Hernán and Tyler J VanderWeele. Compound treatments and transportability of causal inference. Epidemiology (Cambridge, Mass.), 22(3):368, 2011.
-  Manuel Hidalgo, Frederic Amant, Andrew V Biankin, Eva Budinská, Annette T Byrne, Carlos Caldas, Robert B Clarke, Steven de Jong, Jos Jonkers, Gunhild Mari Mælandsmo, et al. Patient-derived xenograft models: an emerging platform for translational cancer research. Cancer discovery, 4(9):998–1013, 2014.
-  Hui Gao, Joshua M Korn, Stéphane Ferretti, John E Monahan, Youzhen Wang, Mallika Singh, Chao Zhang, Christian Schnell, Guizhi Yang, Yun Zhang, et al. High-throughput screening using patient-derived tumor xenografts to predict clinical trial drug response. Nature medicine, 21(11):1318, 2015.