Unbiased Risk Estimators Can Mislead: A Case Study of Learning with Complementary Labels

07/05/2020
by   Yu-Ting Chou, et al.
23

In weakly supervised learning, unbiased risk estimator(URE) is a powerful tool for training classifiers when training and test data are drawn from different distributions. Nevertheless, UREs lead to overfitting in many problem settings when the models are complex like deep networks. In this paper, we investigate reasons for such overfitting by studying a weakly supervised problem called learning with complementary labels. We argue the quality of gradient estimation matters more in risk minimization. Theoretically, we show that a URE gives an unbiased gradient estimator(UGE). Practically, however, UGEs may suffer from huge variance, which causes empirical gradients to be usually far away from true gradients during minimization. To this end, we propose a novel surrogate complementary loss(SCL) framework that trades zero bias with reduced variance and makes empirical gradients more aligned with true gradients in the direction. Thanks to this characteristic, SCL successfully mitigates the overfitting issue and improves URE-based methods.

READ FULL TEXT
research
12/30/2019

Learning from Multiple Complementary Labels

Complementary-label learning is a new weakly-supervised learning framewo...
research
01/13/2020

Multi-Complementary and Unlabeled Learning for Arbitrary Losses and Models

A weakly-supervised learning framework named as complementary-label lear...
research
02/13/2021

Learning from Similarity-Confidence Data

Weakly supervised learning has drawn considerable attention recently to ...
research
03/02/2017

Positive-Unlabeled Learning with Non-Negative Risk Estimator

From only positive (P) and unlabeled (U) data, a binary classifier could...
research
10/20/2019

Mitigating Overfitting in Supervised Classification from Two Unlabeled Datasets: A Consistent Risk Correction Approach

From two unlabeled (U) datasets with different class priors, we can trai...
research
05/14/2013

Estimating or Propagating Gradients Through Stochastic Neurons

Stochastic neurons can be useful for a number of reasons in deep learnin...
research
09/20/2022

Reduction from Complementary-Label Learning to Probability Estimates

Complementary-Label Learning (CLL) is a weakly-supervised learning probl...

Please sign up or login with your details

Forgot password? Click here to reset