Counterfactual Cross-Validation: Effective Causal Model Selection from Observational Data

09/11/2019
by   Yuta Saito, et al.
0

What is the most effective way to select the best causal model among potential candidates? In this paper, we propose a method to effectively select the best individual-level treatment effect (ITE) predictors from a set of candidates using only an observational validation set. In model selection or hyperparameter tuning, we are interested in choosing the best model or the value of hyperparameter from potential candidates. Thus, we focus on accurately preserving the rank order of the ITE prediction performance of candidate causal models. The proposed evaluation metric is theoretically proved to preserve the true ranking of the model performance in expectation and to minimize the upper bound of the finite sample uncertainty in model selection. Consistent with the theoretical result, empirical experiments demonstrate that our proposed method is more likely to select the best model and set of hyperparameter in both model selection and hyperparameter tuning.

READ FULL TEXT
research
11/03/2022

Empirical Analysis of Model Selection for Heterogenous Causal Effect Estimation

We study the problem of model selection in causal inference, specificall...
research
12/20/2022

Out-of-sample scoring and automatic selection of causal estimators

Recently, many causal estimators for Conditional Average Treatment Effec...
research
08/03/2022

Bayesian ranking and selection with applications to field studies, economic mobility, and forecasting

Decision-making often involves ranking and selection. For example, to as...
research
08/24/2019

EPP: interpretable score of model predictive power

The most important part of model selection and hyperparameter tuning is ...
research
07/14/2022

Have we been Naive to Select Machine Learning Models? Noisy Data are here to Stay!

The model selection procedure is usually a single-criterion decision mak...
research
12/08/2020

Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels

For multi-class classification under class-conditional label noise, we p...
research
09/14/2021

Targeted Cross-Validation

In many applications, we have access to the complete dataset but are onl...

Please sign up or login with your details

Forgot password? Click here to reset