Information-theoretic analysis for transfer learning

05/18/2020
by   Xuetong Wu, et al.
0

Transfer learning, or domain adaptation, is concerned with machine learning problems in which training and testing data come from possibly different distributions (denoted as u and u', respectively). In this work, we give an information-theoretic analysis on the generalization error and the excess risk of transfer learning algorithms, following a line of work initiated by Russo and Zhou. Our results suggest, perhaps as expected, that the Kullback-Leibler (KL) divergence D(u||u') plays an important role in characterizing the generalization error in the settings of domain adaptation. Specifically, we provide generalization error upper bounds for general transfer learning algorithms and extend the results to a specific empirical risk minimization (ERM) algorithm where data from both distributions are available in the training phase. We further apply the method to iterative, noisy gradient descent algorithms, and obtain upper bounds which can be easily calculated, only using parameters from the learning algorithms. A few illustrative examples are provided to demonstrate the usefulness of the results. In particular, our bound is tighter in specific classification problems than the bound derived using Rademacher complexity.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/12/2022

An Information-Theoretic Analysis for Transfer Learning: Error Bounds and Applications

Transfer learning, or domain adaptation, is concerned with machine learn...
research
11/14/2022

Renewing Iterative Self-labeling Domain Adaptation with Application to the Spine Motion Prediction

The area of transfer learning comprises supervised machine learning meth...
research
05/31/2023

Hypothesis Transfer Learning with Surrogate Classification Losses

Hypothesis transfer learning (HTL) contrasts domain adaptation by allowi...
research
10/13/2020

Information-Theoretic Bounds on Transfer Generalization Gap Based on Jensen-Shannon Divergence

In transfer learning, training and testing data sets are drawn from diff...
research
11/06/2015

Neutralized Empirical Risk Minimization with Generalization Neutrality Bound

Currently, machine learning plays an important role in the lives and ind...
research
11/02/2021

Characterizing and Understanding the Generalization Error of Transfer Learning with Gibbs Algorithm

We provide an information-theoretic analysis of the generalization abili...
research
05/04/2021

Online Transfer Learning: Negative Transfer and Effect of Prior Knowledge

Transfer learning is a machine learning paradigm where the knowledge fro...

Please sign up or login with your details

Forgot password? Click here to reset