Self-Supervised Training Enhances Online Continual Learning

03/25/2021
by   Jhair Gallardo, et al.
0

In continual learning, a system must incrementally learn from a non-stationary data stream without catastrophic forgetting. Recently, multiple methods have been devised for incrementally learning classes on large-scale image classification tasks, such as ImageNet. State-of-the-art continual learning methods use an initial supervised pre-training phase, in which the first 10 in an offline manner before continual learning of new classes begins. We hypothesize that self-supervised pre-training could yield features that generalize better than supervised learning, especially when the number of samples used for pre-training is small. We test this hypothesis using the self-supervised MoCo-V2 and SwAV algorithms. On ImageNet, we find that both outperform supervised pre-training considerably for online continual learning, and the gains are larger when fewer samples are available. Our findings are consistent across three continual learning algorithms. Our best system achieves a 14.95 the prior state of the art for online continual learning.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/19/2022

Continual Pre-Training Mitigates Forgetting in Language and Vision

Pre-trained models are nowadays a fundamental component of machine learn...
research
07/13/2022

Task Agnostic Representation Consolidation: a Self-supervised based Continual Learning Approach

Continual learning (CL) over non-stationary data streams remains one of ...
research
03/30/2023

Practical self-supervised continual learning with continual fine-tuning

Self-supervised learning (SSL) has shown remarkable performance in compu...
research
09/13/2023

Domain-Aware Augmentations for Unsupervised Online General Continual Learning

Continual Learning has been challenging, especially when dealing with un...
research
12/30/2021

Continually Learning Self-Supervised Representations with Projected Functional Regularization

Recent self-supervised learning methods are able to learn high-quality i...
research
03/13/2021

Online Learning of Objects through Curiosity-Driven Active Learning

Children learn continually by asking questions about the concepts they a...
research
01/17/2022

The CLEAR Benchmark: Continual LEArning on Real-World Imagery

Continual learning (CL) is widely regarded as crucial challenge for life...

Please sign up or login with your details

Forgot password? Click here to reset