Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning

03/22/2021
by   Zheda Mai, et al.
1

Online class-incremental continual learning (CL) studies the problem of learning new classes continually from an online non-stationary data stream, intending to adapt to new data while mitigating catastrophic forgetting. While memory replay has shown promising results, the recency bias in online learning caused by the commonly used Softmax classifier remains an unsolved challenge. Although the Nearest-Class-Mean (NCM) classifier is significantly undervalued in the CL community, we demonstrate that it is a simple yet effective substitute for the Softmax classifier. It addresses the recency bias and avoids structural changes in the fully-connected layer for new classes. Moreover, we observe considerable and consistent performance gains when replacing the Softmax classifier with the NCM classifier for several state-of-the-art replay methods. To leverage the NCM classifier more effectively, data embeddings belonging to the same class should be clustered and well-separated from those with a different class label. To this end, we contribute Supervised Contrastive Replay (SCR), which explicitly encourages samples from the same class to cluster tightly in embedding space while pushing those of different classes further apart during replay-based training. Overall, we observe that our proposed SCR substantially reduces catastrophic forgetting and outperforms state-of-the-art CL methods by a significant margin on a variety of datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/19/2022

Bypassing Logits Bias in Online Class-Incremental Learning with a Generative Framework

Continual learning requires the model to maintain the learned knowledge ...
research
01/25/2021

Online Continual Learning in Image Classification: An Empirical Survey

Online continual learning for image classification studies the problem o...
research
11/10/2022

Mitigating Forgetting in Online Continual Learning via Contrasting Semantically Distinct Augmentations

Online continual learning (OCL) aims to enable model learning from a non...
research
04/09/2021

Unsupervised Class-Incremental Learning Through Confusion

While many works on Continual Learning have shown promising results for ...
research
11/05/2022

Prototypical quadruplet for few-shot class incremental learning

Many modern computer vision algorithms suffer from two major bottlenecks...
research
07/24/2022

Online Continual Learning with Contrastive Vision Transformer

Online continual learning (online CL) studies the problem of learning se...
research
05/25/2023

Condensed Prototype Replay for Class Incremental Learning

Incremental learning (IL) suffers from catastrophic forgetting of old ta...

Please sign up or login with your details

Forgot password? Click here to reset