Characterizing and Understanding the Generalization Error of Transfer Learning with Gibbs Algorithm

11/02/2021
by   Yuheng Bu, et al.
8

We provide an information-theoretic analysis of the generalization ability of Gibbs-based transfer learning algorithms by focusing on two popular transfer learning approaches, α-weighted-ERM and two-stage-ERM. Our key result is an exact characterization of the generalization behaviour using the conditional symmetrized KL information between the output hypothesis and the target training samples given the source samples. Our results can also be applied to provide novel distribution-free generalization error upper bounds on these two aforementioned Gibbs algorithms. Our approach is versatile, as it also characterizes the generalization errors and excess risks of these two Gibbs algorithms in the asymptotic regime, where they converge to the α-weighted-ERM and two-stage-ERM, respectively. Based on our theoretical results, we show that the benefits of transfer learning can be viewed as a bias-variance trade-off, with the bias induced by the source distribution and the variance induced by the lack of target samples. We believe this viewpoint can guide the choice of transfer learning algorithms in practice.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

07/28/2021

Characterizing the Generalization Error of Gibbs Algorithm with Symmetrized KL information

Bounding the generalization error of a supervised learning algorithm is ...
05/18/2020

Information-theoretic analysis for transfer learning

Transfer learning, or domain adaptation, is concerned with machine learn...
10/14/2018

Theoretical Guarantees of Transfer Learning

Transfer learning has been proven effective when within-target labeled d...
06/25/2020

Between-Domain Instance Transition Via the Process of Gibbs Sampling in RBM

In this paper, we present a new idea for Transfer Learning (TL) based on...
05/04/2021

Online Transfer Learning: Negative Transfer and Effect of Prior Knowledge

Transfer learning is a machine learning paradigm where the knowledge fro...
06/30/2021

Learning Bounds for Open-Set Learning

Traditional supervised learning aims to train a classifier in the closed...
06/23/2018

Zeta Distribution and Transfer Learning Problem

We explore the relations between the zeta distribution and algorithmic i...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.