Counterfactual Prediction Under Outcome Measurement Error

02/22/2023
by   Luke Guerdan, et al.
0

Across domains such as medicine, employment, and criminal justice, predictive models often target labels that imperfectly reflect the outcomes of interest to experts and policymakers. For example, clinical risk assessments deployed to inform physician decision-making often predict measures of healthcare utilization (e.g., costs, hospitalization) as a proxy for patient medical need. These proxies can be subject to outcome measurement error when they systematically differ from the target outcome they are intended to measure. However, prior modeling efforts to characterize and mitigate outcome measurement error overlook the fact that the decision being informed by a model often serves as a risk-mitigating intervention that impacts the target outcome of interest and its recorded proxy. Thus, in these settings, addressing measurement error requires counterfactual modeling of treatment effects on outcomes. In this work, we study intersectional threats to model reliability introduced by outcome measurement error, treatment effects, and selection bias from historical decision-making policies. We develop an unbiased risk minimization method which, given knowledge of proxy measurement error properties, corrects for the combined effects of these challenges. We also develop a method for estimating treatment-dependent measurement error parameters when these are unknown in advance. We demonstrate the utility of our approach theoretically and via experiments on real-world data from randomized controlled trials conducted in healthcare and employment domains. As importantly, we demonstrate that models correcting for outcome measurement error or treatment effects alone suffer from considerable reliability limitations. Our work underscores the importance of considering intersectional threats to model validity during the design and evaluation of predictive models for decision support.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/05/2023

Improving Image-Based Precision Medicine with Uncertainty-Aware Causal Models

Image-based precision medicine aims to personalize treatment decisions b...
research
08/30/2019

Counterfactual Risk Assessments, Evaluation, and Fairness

Algorithmic risk assessments are increasingly used to help humans make d...
research
09/15/2022

Decision making in cancer: Causal questions require causal answers

Treatment decisions in cancer care are guided by treatment effect estima...
research
03/01/2020

When the Oracle Misleads: Modeling the Consequences of Using Observable Rather than Potential Outcomes in Risk Assessment Instruments

Machine learning-based Risk Assessment Instruments are increasingly wide...
research
09/18/2023

Beyond expected values: Making environmental decisions using value of information analysis when measurement outcome matters

In ecological and environmental contexts, management actions must someti...
research
09/21/2023

Uplift vs. predictive modeling: a theoretical analysis

Despite the growing popularity of machine-learning techniques in decisio...

Please sign up or login with your details

Forgot password? Click here to reset