Empirical Analysis of Model Selection for Heterogenous Causal Effect Estimation

11/03/2022
by   Divyat Mahajan, et al.
0

We study the problem of model selection in causal inference, specifically for the case of conditional average treatment effect (CATE) estimation under binary treatments. Unlike model selection in machine learning, we cannot use the technique of cross-validation here as we do not observe the counterfactual potential outcome for any data point. Hence, we need to design model selection techniques that do not explicitly rely on counterfactual data. As an alternative to cross-validation, there have been a variety of proxy metrics proposed in the literature, that depend on auxiliary nuisance models also estimated from the data (propensity score model, outcome regression model). However, the effectiveness of these metrics has only been studied on synthetic datasets as we can observe the counterfactual data for them. We conduct an extensive empirical analysis to judge the performance of these metrics, where we utilize the latest advances in generative modeling to incorporate multiple realistic datasets. We evaluate 9 metrics on 144 datasets for selecting between 415 estimators per dataset, including datasets that closely mimic real-world datasets. Further, we use the latest techniques from AutoML to ensure consistent hyperparameter selection for nuisance models for a fair comparison across metrics.

READ FULL TEXT
research
08/29/2020

Model selection for estimation of causal parameters

A popular technique for selecting and tuning machine learning estimators...
research
09/11/2019

Counterfactual Cross-Validation: Effective Causal Model Selection from Observational Data

What is the most effective way to select the best causal model among pot...
research
02/01/2023

How to select predictive models for causal inference?

Predictive models – as with machine learning – can underpin causal infer...
research
04/30/2015

Model Selection and Overfitting in Genetic Programming: Empirical Study [Extended Version]

Genetic Programming has been very successful in solving a large area of ...
research
12/20/2022

Out-of-sample scoring and automatic selection of causal estimators

Recently, many causal estimators for Conditional Average Treatment Effec...
research
03/02/2023

Hyperparameter Tuning and Model Evaluation in Causal Effect Estimation

The performance of most causal effect estimators relies on accurate pred...

Please sign up or login with your details

Forgot password? Click here to reset