Policy decisions are often informed by aggregate information on cost and clinical effectiveness. Several measures have been previously proposed to compare the cost-effectiveness of two interventions. Arguably, the simplest of these measures is the incremental cost-effectiveness ratio (ICER), given by the difference in mean cost divided by the difference in mean clinical measure [1, 2, 3]. When the ICER lies below a pre-determined willingness-to-pay (WTP) threshold, the experimental treatment is deemed cost-effective. The net monetary benefit (NMB) is an equivalent measure to the ICER that was proposed to avoid a singularity in the denominator, as well as known challenges associated with interval estimation of ratio parameters [4, 5]. The NMB has been used to compare cost-effectiveness in many recent studies [6, 7, 8, 9, 10].
The cost-effectiveness acceptability (CEA) curve was proposed as a probabilistic representation of strength of evidence regarding NMB; its function is to elucidate how conclusions vary with changes in the WTP 
. The CEA is defined as the proportion of positive bootstrapped estimates of the NMB; it is sometimes loosely referred to as the ”probability of cost-effectiveness,”[12, 13, 14] but more closely resembles a frequentist p-value . The CEA has been used as a primary tool to advocate policy changes. Delaney et al., for instance, utilize a CEA as evidence in support of initial endoscopy in dyspepsia patients , and mental health studies have commonly included CEAs for decision-making purposes in various interventions, sometimes under the name ”probabilistic sensitivity analysis” [17, 18]. However, Barton et al. note that the CEA, despite its widespread use, does not alone provide a path for optimal decision-making as it does not provide insights into the extent of cost-effectiveness .
Some have proposed alternative graphical tools to the CEA in response to the challenges associated with interpreting it as a probability of cost-effectiveness within the frequentist paradigm. One approach, proposed by Heitjan et al., was to re-frame the NMB and acceptability curve within a Bayesian paradigm so that a probabilistic interpretation would be warranted [20, 21]. Hoch and Blume present a likelihood ratio approach based on a pre-specified alternative hypothesis . These approaches, like the CEA, are still guided by inference and strength of evidence regarding the NMB, quantifying either the degree of belief regarding the mean cost-effectiveness a posteriori (Bayesian methods), or the strength of evidence in support of a positive NMB (likelihood methods).
While the CEA provides insights into sensitivity of conclusions regarding mean clinical/cost outcome combinations to fluctuations in the WTP, a probabilistic measure of cost-effectiveness magnitude—i.e., a parameter that quantifies the frequency with which treatment produces more desirable clinical/cost outcome combinations—can summarize the distribution of the cost/clinical outcomes as a whole in a way that complements insights from mean differences. In this paper, we define the net benefit separation (NBS) as the probability that an observation from a hypothetical population in which all patients receive treatment has greater individual net benefit than an observation from a hypothetical population in which all patients receive the control. This gives rise to a visualization tool in which the NBS is plotted over a range of willingness-to-pay thresholds, which we term the cost-effectiveness determination (CED) curve. In this paper, we will further demonstrate how weighting and standardization can be used to estimate the NBS and its corresponding CED curve when outcomes are right-censored and the data are subject to confounding.
The remainder of this paper is organized as follows. In Section 2, we provide a review of the net monetary benefit and its associated cost-effectiveness acceptability curve. In Section 3, we define the net benefit separation and its associated determination curve. In Section 4, we propose an estimation procedure to estimate net benefit separation using observational data. In Section 5, we present results from a simulation study in order to understand the finite-sample properties of our methods. In Section 6, we present an illustration of these methods using simulation-based data from a prior study of endometrial cancer patients . We conclude with a discussion of our findings in Section 7.
2 Review of the Net Monetary Benefit and Acceptability Curve
2.1 The net monetary benefit
Let index independently sampled study participants, and let denote binary treatment. Let denote the clinical measure (i.e., the outcome that determines the treatment’s effectiveness), and the cost. For example, could be systolic blood pressure reduction or survival time, and could be initial treatment cost or total medical cost billed through some time range. The net monetary benefit (NMB) is defined as follows:
Here, denotes the WTP and is defined as the maximum cost one would be willing to pay for a one-unit improvement in associated with treatment (for example, $50,000 for year of life saved). Note that indicates treatment is more cost-effective on average, while indicates the reverse.
2.2 The cost-effectiveness acceptability curve
The CEA is defined according to the following procedure:
Define some range of possible willingness-to-pay thresholds.
Draw a full-sized bootstrap replicate from the set .
Compute using the bootstrapped data for each .
For , the acceptability curve is defined as:
While sometimes stated as the probability of cost-effectiveness [12, 13, 23, 14, 11], this is more accurately described as one minus a one-sided bootstrapped p-value of the lesser alternative that the control is more cost-effective on average. The limiting behavior of the CEA can be described as follows. First fix : at , clinical effects are not considered, and as increases, the mean cost difference wanes in its influence on . Now consider asymptotics in , and suppose first that the difference in mean clinical outcome is nonzero (the non-degenerate case). As grows, the CEA converges point-wise to a step function:
If instead the difference in mean clinical outcome is zero, then
In the non-degenerate case, the limit of the CEA informs the value of at which the treatments are equally cost-effective on average (i.e., at the ICER); equivalently, it informs the values of at which the NMB is positive, regardless of magnitude. Figure 1 illustrates the CEA’s limiting behavior when treatment produces higher mean cost and clinical benefit.
For a given , a higher CEA value indicates stronger support in favor of cost-effectiveness for treatment
; however, values closer to zero do not distinguish between ”absence of evidence for the experimental treatment’s cost-effectiveness” and ”evidence of cost-effectiveness for the control treatment.” This challenge regarding lack of symmetry can be avoided by instead plotting the estimated NMB and confidence intervals across a range of. This graphical representation accomplishes the goal of the CEA (which is to provide insights into the sensitivity of conclusions regarding the value of the NMB to changes in ) with the added advantage of depicting both the point and interval estimates.
3 Net Benefit Separation: A Probabilistic Approach
Recall that denotes the clinical/cost outcome pair. Define to be the individual net benefit (INB). Following the potential outcomes notation , let denote the clinical and cost outcome pair under treatment , and the INB under treatment . One straightforward probabilistic metric of cost-effectiveness is defined at the subject specific level: . This can be described as the probability that the treatment is cost-effective for a random subject. Because only one potential outcome pair can be observed per subject, is not identifiable without imposing unrealistic and untestable assumptions (e.g., uncorrelatedness of and ).
We instead propose a measure of stochastic ordering of the marginal distributions under each hypothetical comparator treatment. To avoid notational confusion, let denote a randomly sampled value for under the setting in which everyone receives treatment . We define the net benefit separation (NBS) by
. In essence, the NBS targets the area under the receiver operating characteristic curve, frequently used in the evaluation of diagnostic measures, and is so named as it takes on the value one only when the distribution of INBs understochastically dominates the distribution under , and zero if the reverse is true. The NBS takes on the value of one-half, for instance, when the treatment has null clinical and cost effects (Figure 2).
3.1 The case for considering both the NMB and NBS
The NMB and NBS can be seen as complementary measures of cost-effectiveness (the former is a comparison of population means, while the latter is a measure of stochastic ordering). It is possible for the two to provide seemingly contradictory stories at times. In particular, . Particularly since it is not clear that one of these measures is uniformly more informative than the other, this provides a case for considering both NMB and NBS in a cost-effectiveness analysis.
Imagine a scenario in which , but
. This can happen, for instance, when there is a high level of right-skewness that drives a mean difference. In such a setting, one may put more weight onin regards to decisions on whether to apply the experimental treatment to the entire population. On the other hand, imagine a scenario in which is very high, but that . This can happen, for instance, when a large segment of the population exhibits a modest level of treatment benefit with very high precision (i.e., low variability).
These examples, while hypothetical, underscore the tension between scientific questions pertaining to whether a treatment tends to produce more desirable outcomes on average and scientific questions pertaining to whether a treatment tends to produce more desirable outcomes in probability. Evidence of discordance between and can spark further exploration in order to learn about possible sources for heterogeneity.
3.2 The determination curve and its limiting behavior
The limiting behavior of can be characterized as follows. We have that and , where denotes a randomly sampled clinical and cost outcome pair under treatment . We define the cost-effectiveness determination (CED) curve as , where again denotes a reasonable set of WTP threshold values.
4.1 Randomized trials with no censoring
To orient our discussion of estimation, we first consider the idealized setting of a randomized trial in which there is no censoring (this can equivalently be thought of as a randomized trial that considers a non-survival clinical measure). A nonparametric estimate of can be constructed in this case: letting index the patients in the control arm and index the patients in the experimental arm,
is trivially unbiased and consistent for . It is perhaps better expressed using the equivalent and less computationally taxing scaled variant of the Wilcoxon Rank-Sum statistic:
where denotes the rank of the the pooled INBs. In this setting, the sets and each form nonparametric estimates of the marginal distributions for and , respectively. The remainder of this section is therefore geared toward approaches for estimating those marginal distributions under more complex (but more realistic) settings in which there is confounding and survival time is the clinical outcome.
4.2 Semiparametric standardization
Suppose we are in the setting of observational data. Letting L denote (observed) confounders, the NBS can be expressed as:
We may then employ Monte-Carlo standardization for estimation:
Define some range of possible willingness-to-pay thresholds.
Estimate the distribution of L (e.g., via the empirical ).
Fit models for and .
For each :
Generate random draws from .
Generate random draws: from and from , as estimated in Step 3.
Compute the ranks , using the pooled data, of the INBs from the realizations drawn from .
For each , an estimate the net benefit separation is given by:
This method specifies parametric models for the conditional distributions ofand , but not for the distribution of the confounders. Importantly, the parametric models can be quite flexible, easily accommodating regression splines and zero-inflation of cost using a two-part model .
4.3 Addressing censoring and truncated cost
Suppose now that is survival time. We further assume that is considered over time range . Let denote the censoring time; in practice, is observed. Let denote the indicator of censored survival time, and the indicator of censored costs. Inverse probability-of-censoring weights (IPCW) must be included to estimate owing to informative censoring induced by heterogeneity in cost-accrual rates [26, 27, 28]. Weights can be obtained via the Kaplan-Meier method. Let , and the Kaplan-Meier (KM) estimator based on . If covariates are discrete, stratification is possible; for other cases (e.g., continuous covariates), we may instead use employ the stratified Cox proportional hazards model :
where V denotes stratification variables and W other covariates. Here, denotes the hazard function for censoring, and we assume that ; covariates in V and W may include those in L. Letting , the probability of being observed can be estimated by:
Note that denotes the partial likelihood estimator .
4.4 Interval estimation and inference
Confidence intervals can be formed using the nonparametric bootstrap . In the case of Section 4.1.,
under the null hypothesis
and equal variances, wheredenotes the randomization ratio . This may be used for asymptotic inference. Otherwise, hypothesis testing can also be conducted using the nonparametric bootstrap.
5 Simulation Study
We conduct a simulation study to evaluate the performance of under two sample sizes ( and 5,000) and two censoring rates (10% and 25%). Covariates are simulated as and , and treatment is given by . Survival is generated as ), and censoring is generated as . We generate the cost such that . The parameter controls the censoring rate and is set to for 10% censoring (low) and for 25% censoring (high). We consider two simulation scenarios. The first is a null scenario in which the treatment has no impact on cost or survival ( and ); the second is such that treatment is associated with an increase in both cost and survival (, , and ). We consider and as examples.
For each case, we conduct one-thousand replicates with bootstrap replicates and 5,000 Monte-Carlo iterations. The distribution of was estimated nonparametrically using the empirical distribution for the standardization procedure. The Kaplan-Meier estimator is used to determine the IPCW weights, stratified by . We report the mean estimate
across simulations, as well as the average estimated standard error and empirical standard error across Monte-Carlo replicates.
Tables 1 and 2 present results for each scenario. The proposed procedure is able to estimate with low bias, and the nonparametric bootstrap standard error adequately represents the true repeat-sample variability. For lower samples sizes, the bootstrap standard error slightly underestimates the true standard error on average for lower values of , and slightly overestimate the standard error for higher values of . This bias dissipates with a larger sample size.
We conduct a cost-effectiveness analysis using a simulation-based data set motivated from a prior study of survival and total medical costs in post-hysterectomy stage-I and stage-II endometrial cancer patients . We fit a sequence of flexible models on the variables from the original database and simulated a data set of equal size from the estimated models. Our reasons for taking this approach are twofold: (1) doing so allows us to make the data set fully available; we would not be able to do so otherwise due to the proprietary nature of the data, and (2) the proportion of individuals having an event by the study’s end was low due to insufficient follow-up time. Hence, outcomes were simulated over a ten-year window using an aggregate of the original data and estimates from the literature [32, 33]. We provide the simulated data set and code in the supplementary materials. Results are intended solely for illustration purposes.
Post-surgery, endometrial cancer patients may receive adjuvant treatment with radiation therapy (RT), chemotherapy (CT), or neither . Though there are apparent survival advantages associated with adjuvant RT and CT, little has been done to elucidate the cost-effectiveness of either. Our illustration focuses on determining cost-effectiveness associated with each of adjuvant RT alone and adjuvant CT alone received two to four months post-surgery relative to control.
6.1 Description of data and analysis
Covariates include age of diagnosis, Charlson co-morbidity index , number of prior hospitalizations, cancer stage, and prior receipt of RT or CT. These data include 13,526 patients. The mean age of diagnosis was 73.7 years (SD = 6.59 years), and 93.6% were diagnosed with stage I cancer. At baseline, 5.99% of patients had received RT and 0.55% had received CT. Through the two-to-four month window, 25.4% received RT and 2.25% received CT (0.79% received both RT and CT). A total of 7,720 deaths occurred (57.0%) and 8,762 patients (64.7%) had complete cost data.
We employ the procedure outlined in Section 4.2 and incorporate weights from the Cox model as outlined in Section 4.3. We utilize the empirical joint distribution of the baseline covariates, and incorporate each covariate as a predictor of censoring time. We fit a parametric Weibull model to the survival distribution conditional on covariates and treatment (each of RT and CT). We estimate the distribution of costs using a two-part zero-inflation model (a logistic model for the probability of zero-costs, followed by a log-normal model for nonzero costs). Quantiles from1,000 bootstrap replicates were used to formulate 95% confidence intervals, and 10,000 replicates were used for the Monte-Carlo draws associated with our standardization procedure.
Our target parameters include and , which are defined by comparing: (1) RT without CT to control, and (2) CT without RT to control, respectively (where control is defined as no RT or CT). We additionally target and , which correspond to the net monetary benefit based on comparisons (1) and (2). We primarily consider a range of 50 to 120 for (in units of thousands of USD over one year), though we extend the range in order to illustrate limiting behavior. In our analysis, the NMB is determined using the Monte-Carlo draws from the standardization procedure.
Table 3 presents the results of our analysis for the boundary values of under primary consideration. As an example of how to interpret the estimated NBS, , consider a WTP threshold of $120,000, and imagine two hypothetical situations: one in which the population receives CT (but no RT), and another in which the population does not receive adjuvant RT or CT. We estimate that the INB from the individual sampled from the RT population would be higher 67% of the time upon repeated random sampling from these two hypothetical scenarios, with a 95% CI of (63%, 72%). Figure 3 presents the CED curves for each comparison. Note that and are both smaller than 0.5, corresponding to higher costs in the treatment groups. At the lowest value of the WTP used for primary consideration ($50,000), these results suggest cost-effectiveness for both RT and CT as compared to control.
Table 3 additionally presents the estimated NMB at the same values of along with 95% confidence intervals. Figure 4 depicts the estimated NMB, with confidence intervals included for the primary range of . Similarly to the NBS, the confidence intervals do not include zero, suggesting evidence of cost-effectiveness as measured by mean differences. Figure 5 depicts corresponding CEA curves, which suggest strong evidence that both RT and CT are cost-effective relative to control on average for . Moreover, the strength of evidence is not sensitive to changes in over the range of interest (note this would not be so for lower values of ). These insights could also be gleaned from examination of Figure 4.
In this example, we find that the results from the NBS are consistent with those of the NMB, suggesting that both RT and CT are cost-effective relative to control.
In this paper, we have first summarized the net monetary benefit and its corresponding acceptability curve, both of which are standard in cost-effectiveness analysis. We stated the proper interpretation of results from these approaches and underscored some of the limitations associated with current methodology.
Our proposed novel metric, the NBS, is a probabilistic population parameter that quantifies the magnitude of cost-effectiveness, having an interpretation similar in spirit to that of the area under a receiver operating characteristic curve in diagnostic testing. Our approach to estimation can accommodate challenges typically encountered in observational studies such as confounding and censoring via standardization and inverse probability weighting, respectively.
The limitations of the acceptability curve that we have delineated do not generally apply to the net monetary benefit as a whole. The net monetary benefit is a summary of average cost-effectiveness and is useful in its own right. In order to graphically understand how the magnitude of the net monetary benefit and evidence of a non-zero average effect on cost-effectiveness change across the willingness-to-pay, we demonstrate that one may plot the estimated net monetary benefits and respective confidence intervals across values of with no need to consider the acceptability curve.
In Section 3.1, we describe scenarios in which the NMB and NBS could provide seemingly contradicting information. This potential dichotomy cannot be resolved without fully understanding the context of the treatment and disease setting, including each treatment’s safety profile and the clinical outcomes under consideration. It also certainly cannot be resolved unless both measures are computed and understood. A discordance between the two metrics would likely prompt subgroup analyses based on important variables to try to reconcile the systematic sources of heterogeneity; we are currently developing approaches for subgroup analysis in cost-effectiveness.
One potential advantage of the net monetary benefit is the well-ordering/transitivity property associated with simultaneous comparison of multiple competing treatment regimes. That is, if treatment is better than treatment and treatment is better than treatment , then treatment will be better than treatment when ”better” means a higher NMB. It is known that transitivity does not always hold with measures of stochastic ordering like the NBS. However, the NBS satisfies the previously unaddressed methodological gap pertaining to the lack of a sensible probabilistic measure of cost-effectiveness magnitude.
Issues surrounding unobserved confounding apply to both the NMB and the NBS approach, and always merits close scrutiny in observational data. The risk of bias due to unobserved confounding is always real in observational data, although care can be taken in many cases at the data collection stage in order to to mitigate the impact. Approaches to assess sensitivity of results to the assumption of no unmeasured confounding would be of interest in this setting .
Our application is designed to illustrate a properly conducted cost-effectiveness analysis—one in which insights regarding both the net monetary benefit and net benefit separation are aggregated to paint a complete picture regarding treatment’s cost-effectiveness. Importantly, our method for determining the NMB in this example is distinct from that employed in prior literature. Li et al.  propose IPCW and propensity-score weights in order to estimate the NMB; this approach is based on restricted mean survival time, in which the restriction is to the upper bound on censoring time. Since our parametric approach for estimating the survival distribution does not demand restriction, considering the semi-parametric approach of Li et al. would induce an incongruity in a comparison of the two cost-effectiveness methods.
-  Polsky D, Glick H, Willke R et al. Confidence intervals for cost-effectiveness ratios: a comparison of four methods. Health Economics 1997; 6(3): 243–252.
-  Willan A and Briggs A. Statistical Analysis of Cost-Effectiveness Data, volume 37. John Wiley & Sons, 2006.
-  Willan A and O’Brien B. Confidence intervals for cost-effectiveness ratios: an application of Fieller’s theorem. Health Economics 1996; 5(4): 297–305.
-  Briggs A, Wonderling D and Mooney C. Pulling cost-effectiveness analysis up by its bootstraps: A non-parametric approach to confidence interval estimation. Health Economics 1997; 6(4): 327–340.
-  Heitjan D, Moskowitz A and Whang W. Problems with interval estimates of the incremental cost-effectiveness ratio. Medical Decision Making 1999; 19: 9–15.
-  Leppert M, Poisson S, Carroll J et al. Cost-effectiveness of patent foramen ovale closure versus medical therapy for secondary stroke prevention. Stroke 2018; 49: 1443–1450.
-  Liao C, Tan E, Chen C et al. Real-world cost-effectiveness of laparoscopy versus open colectomy for colon cancer: a nationwide population-based study. Surgical Endoscopy 2017; 31: 1796–1805.
-  Zhang T, Xiang J, Cui B et al. Cost-effectiveness analysis of fecal microbiota transplantation for inflammatory bowel disease. Oncotarget 2017; 8: 88894–88903.
-  Maru S, Byrnes J, Carrington M et al. Cost-effectiveness of home versus clinic-based management of chronic heart failure: Extended follow-up of a pragmatic, multicentre randomized trial cohort - the WHICH? study (which heart failure intervention is most cost-effective & consumer friendly in reducing hospital care). International Journal of Cardiology 2015; 201: 368–375.
-  Shafrin J, Skornicki M, Brauer M et al. An exploratory case study of the impact of expanding cost-effectiveness analysis for second-line nivolumab for patients with squamous non-small cell lung cancer in canada: Does it make a difference? Health Policy 2018; 122: 607–613.
-  M L and N Z. Definition, interpretation and calculation of cost-effectiveness acceptability curves. Health Economics 2000; 9.
-  Briggs A. Probabilistic analysis of cost-effectiveness models: statistical representation of parameter uncertainty. Value in Health 2005; 8: 1–2.
-  Fenwick E, O’Brien B and Briggs A. Cost-effectiveness acceptability curves–facts, fallacies and frequently asked questions. Health Economics 2004; 13: 405–415.
-  Fenwick E, Marshall D, Levy A et al. Using and interpreting cost-effectiveness acceptability curves: an example using data from a trial of management strategies for atrial fibrillation. BMC Health Services Research 2006; 6.
-  Hoch J and Blume J. Problems with interval estimates of the incremental cost—effectiveness ratio. Journal of Health Economics 2008; 27: 476–495.
-  Delaney B, Wilson S, Roalfe A et al. Cost effectiveness of initial endoscopy for dyspepsia in patients over age 50 years: a randomised controlled trial in primary care. The Lancet 2000; 356(9246): 1965–1969.
-  Ahern S, Burke L, McElroy B et al. A cost-effectiveness analysis of school-based suicide prevention programmes. European Child & Adolescent Psychiatry 2018; 27: 1295–1304.
-  Hunter R, Nazareth I, Morris S et al. Modelling the cost-effectiveness of preventing major depression in general practice patients. Psychological Medicine 2014; 44: 1381–1390.
-  Baron G, Briggs A and EA F. Optimal cost-effectiveness decisions: the role of the cost-effectiveness acceptability curve (ceac), the cost-effectiveness acceptability frontier (ceaf), and the expected value of perfection information (evpi). Value Health 2008; 13: 886–897.
-  Heitjan D, Moskowitz A and Whang W. Bayesian estimation of cost-effectiveness ratios from clinical trials. Health Economics 1999; 8(3): 191–201.
-  Heitjan D, Kim C and Li H. Bayesian estimation of cost-effectiveness from censored data. Statistics in Medicine 2004; 23: 886–897.
-  Spieker A, Ko E, Roy J et al. Nested g-computation: A causal approach to analysis of censored medical costs in the presence of time-varying treatment. Under review; arXiv preprint arXiv:170508742 2019; .
-  Li J, Vachani A, Epstein A et al. A doubly robust approach for cost–effectiveness estimation from observational data. Statistical Methods in Medical Research 2018; 27(10): 3126–3138.
-  Rubin D. Bayesian inference for causal effects: The role of randomization. The Annals of Statistics 1978; 6(1): 34–58. DOI:101214/aos/1176344064.
-  Leung S and Yu S. On the choice between sample selection and two-part models. Journal of Econometrics 1996; 72(1): 197–229.
-  Spieker A, Roy J and Mitra N. Analyzing medical costs with time-dependent treatment: The nested g-formula. Health Economics 2018; 27(7): 1063–1073.
Linear regression analysis of censored medical costs.Biostatistics 2000; 1(1): 35–47.
-  Li J, Handorf E, Bekelman J et al. Propensity score and doubly robust methods for estimating the effect of treatment on censored cost. Statistics in Medicine 2016; 35(12): 1985–1999.
-  Cox D. Regression models and life-tables. Journal of the Royal Statistical Society, Series B 1972; 34: 187–220.
-  Cox D. Partial likelihood. Biometrika 1975; 62: 269–276.
-  Davison A and Hinkley D. Bootstrap Methods and Their Application. New York, NY: Cambridge University Press, 1997.
-  Shaeffer D and Randall M. Adjuvant radiotherapy in endometrial carcinoma. The Oncologist 2008; 10: 623–631.
-  Susumu N, Sagae S, Udagawa Y et al. Randomized phase III trial of pelvic radiotherapy versus cisplatin-based combined chemotherapy in patients with intermediate-and high-risk endometrial cancer: a japanese gynecologic oncology group study. Gynecologic Oncology 2008; 108: 226–233.
-  Braun M, Overbeek-Wager E and Grumbo R. Diagnosis and management of endometrial cancer. American Family Physician 2016; 93.
-  Charlson M, TP S, Peterson J et al. Validation of a combined comorbidity index. Journal of Clinical Epidemiology 2016; 47: 1245–1251.
-  Handorf E, Bekelman J, Heitjan D et al. Evaluating costs with unmeasured confounding: A sensitivity analysis for the treatment effect. The Annals of Applied Statistics 2013; 7(4): 2062–2080.