DeepAI
Log In Sign Up

Sharp Asymptotics of Self-training with Linear Classifier

05/16/2022
by   Takashi Takahashi, et al.
0

Self-training (ST) is a straightforward and standard approach in semi-supervised learning, successfully applied to many machine learning problems. The performance of ST strongly depends on the supervised learning method used in the refinement step and the nature of the given data; hence, a general performance guarantee from a concise theory may become loose in a concrete setup. However, the theoretical methods that sharply predict how the performance of ST depends on various details for each learning scenario are limited. This study develops a novel theoretical framework for sharply characterizing the generalization abilities of the models trained by ST using the non-rigorous replica method of statistical physics. We consider the ST of the linear model that minimizes the ridge-regularized cross-entropy loss when the data are generated from a two-component Gaussian mixture. Consequently, we show that the generalization performance of ST in each iteration is sharply characterized by a small finite number of variables, which satisfy a set of deterministic self-consistent equations. By numerically solving these self-consistent equations, we find that ST's generalization performance approaches to the supervised learning method with a very simple regularization schedule when the label bias is small and a moderately large number of iterations are used.

READ FULL TEXT

page 1

page 2

page 3

page 4

06/19/2020

Statistical and Algorithmic Insights for Semi-supervised Learning with Self-training

Self-training is a classical approach in semi-supervised learning which ...
05/03/2017

Semi-supervised cross-entropy clustering with information bottleneck constraint

In this paper, we propose a semi-supervised clustering method, CEC-IB, t...
05/16/2021

Semi-supervised Contrastive Learning with Similarity Co-calibration

Semi-supervised learning acts as an effective way to leverage massive un...
10/03/2021

Information-Theoretic Generalization Bounds for Iterative Semi-Supervised Learning

We consider iterative semi-supervised learning (SSL) algorithms that ite...
05/15/2019

ROI Regularization for Semi-supervised and Supervised Learning

We propose ROI regularization (ROIreg) as a semi-supervised learning met...