Transfer without Forgetting

06/01/2022
by   Matteo Boschini, et al.
0

This work investigates the entanglement between Continual Learning (CL) and Transfer Learning (TL). In particular, we shed light on the widespread application of network pretraining, highlighting that it is itself subject to catastrophic forgetting. Unfortunately, this issue leads to the under-exploitation of knowledge transfer during later tasks. On this ground, we propose Transfer without Forgetting (TwF), a hybrid Continual Transfer Learning approach building upon a fixed pretrained sibling network, which continuously propagates the knowledge inherent in the source domain through a layer-wise loss term. Our experiments indicate that TwF steadily outperforms other CL methods across a variety of settings, averaging a 4.81 Class-Incremental accuracy over a variety of datasets and different buffer sizes.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/21/2019

Continual Learning with Adaptive Weights (CLAW)

Approaches to continual learning aim to successfully learn a set of rela...
research
07/09/2021

Continual Learning in the Teacher-Student Setup: Impact of Task Similarity

Continual learning-the ability to learn many tasks in sequence-is critic...
research
10/25/2021

Mixture-of-Variational-Experts for Continual Learning

One significant shortcoming of machine learning is the poor ability of m...
research
03/16/2022

Continuous Detection, Rapidly React: Unseen Rumors Detection based on Continual Prompt-Tuning

Since open social platforms allow for a large and continuous flow of unv...
research
05/05/2021

Continual Learning on the Edge with TensorFlow Lite

Deploying sophisticated deep learning models on embedded devices with th...
research
09/16/2020

Measuring Information Transfer in Neural Networks

Estimation of the information content in a neural network model can be p...
research
11/03/2018

Closed-Loop GAN for continual Learning

Sequential learning of tasks using gradient descent leads to an unremitt...

Please sign up or login with your details

Forgot password? Click here to reset