Stable Learning via Causality-based Feature Rectification

by   Zhengxu Yu, et al.

How to learn a stable model under agnostic distribution shift between training and testing datasets is an essential problem in machine learning tasks. The agnostic distribution shift caused by data generation bias can lead to model misspecification and unstable performance across different test datasets. Most of the recently proposed methods are causality-based sample reweighting methods, whose performance is affected by sample size. Moreover, these works are restricted to linear models, not to deep-learning based nonlinear models. In this work, we propose a novel Causality-based Feature Rectification (CFR) method to address the model misspecification problem under agnostic distribution shift by using a weight matrix to rectify features. Our proposal based on the fact that the causality between stable features and the ground truth is consistent under agnostic distribution shift, but is partly omitted and statistically correlated with other features. We propose the feature rectification weight matrix to reconstruct the omitted causality by using other features as proxy variables. We further propose an algorithm that jointly optimizes the weight matrix and the regressor (or classifier). Our proposal can not only improve the stability of linear models, but also deep-learning based models. Extensive experiments on both synthetic and real-world datasets demonstrate that our proposal outperforms previous state-of-the-art stable learning methods. The code will be released later on.


page 1

page 2

page 3

page 4


Stable Prediction across Unknown Environments

In many important machine learning applications, the training distributi...

Prediction with Incomplete Data under Agnostic Mask Distribution Shift

Data with missing values is ubiquitous in many applications. Recent year...

Balance-Subsampled Stable Prediction

In machine learning, it is commonly assumed that training and test data ...

Stable Prediction with Model Misspecification and Agnostic Distribution Shift

For many machine learning algorithms, two main assumptions are required ...

Stable Learning via Sparse Variable Independence

The problem of covariate-shift generalization has attracted intensive re...

Estimating Granger Causality with Unobserved Confounders via Deep Latent-Variable Recurrent Neural Network

Granger causality analysis, as one of the most popular time series causa...

Task Agnostic and Post-hoc Unseen Distribution Detection

Despite the recent advances in out-of-distribution(OOD) detection, anoma...

Please sign up or login with your details

Forgot password? Click here to reset