SPeCiaL: Self-Supervised Pretraining for Continual Learning

06/16/2021
by   Lucas Caccia, et al.
8

This paper presents SPeCiaL: a method for unsupervised pretraining of representations tailored for continual learning. Our approach devises a meta-learning objective that differentiates through a sequential learning process. Specifically, we train a linear model over the representations to match different augmented views of the same image together, each view presented sequentially. The linear model is then evaluated on both its ability to classify images it just saw, and also on images from previous iterations. This gives rise to representations that favor quick knowledge retention with minimal forgetting. We evaluate SPeCiaL in the Continual Few-Shot Learning setting, and show that it can match or outperform other supervised pretraining approaches.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 3

page 6

07/26/2021

Continual-wav2vec2: an Application of Continual Learning for Self-Supervised Automatic Speech Recognition

We present a method for continual learning of speech representations for...
06/28/2021

Co^2L: Contrastive Continual Learning

Recent breakthroughs in self-supervised learning show that such algorith...
06/26/2020

Bookworm continual learning: beyond zero-shot learning and continual learning

We propose bookworm continual learning(BCL), a flexible setting where un...
02/21/2020

Learning to Continually Learn

Continual lifelong learning requires an agent or model to learn many seq...
01/28/2021

Generalising via Meta-Examples for Continual Learning in the Wild

Learning quickly and continually is still an ambitious task for neural n...
02/11/2021

Reproducibility Report: La-MAML: Look-ahead Meta Learning for Continual Learning

The Continual Learning (CL) problem involves performing well on a sequen...
12/30/2021

Continually Learning Self-Supervised Representations with Projected Functional Regularization

Recent self-supervised learning methods are able to learn high-quality i...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.