Initial Classifier Weights Replay for Memoryless Class Incremental Learning

by   Eden Belouadah, et al.

Incremental Learning (IL) is useful when artificial systems need to deal with streams of data and do not have access to all data at all times. The most challenging setting requires a constant complexity of the deep model and an incremental model update without access to a bounded memory of past data. Then, the representations of past classes are strongly affected by catastrophic forgetting. To mitigate its negative effect, an adapted fine tuning which includes knowledge distillation is usually deployed. We propose a different approach based on a vanilla fine tuning backbone. It leverages initial classifier weights which provide a strong representation of past classes because they are trained with all class data. However, the magnitude of classifiers learned in different states varies and normalization is needed for a fair handling of all classes. Normalization is performed by standardizing the initial classifier weights, which are assumed to be normally distributed. In addition, a calibration of prediction scores is done by using state level statistics to further improve classification fairness. We conduct a thorough evaluation with four public datasets in a memoryless incremental learning setting. Results show that our method outperforms existing techniques by a large margin for large-scale datasets.



There are no comments yet.


page 1

page 2

page 4


ScaIL: Classifier Weights Scaling for Class Incremental Learning

Incremental learning is useful if an AI agent needs to integrate data fr...

Class-incremental Learning using a Sequence of Partial Implicitly Regularized Classifiers

In class-incremental learning, the objective is to learn a number of cla...

A Simple Class Decision Balancing for Incremental Learning

Class incremental learning (CIL) problem, in which a learning agent cont...

Class-incremental Learning with Pre-allocated Fixed Classifiers

In class-incremental learning, a learning agent faces a stream of data w...

Dataset Knowledge Transfer for Class-Incremental Learning without Memory

Incremental learning enables artificial agents to learn from sequential ...

Improving Vision Transformers for Incremental Learning

This paper studies using Vision Transformers (ViT) in class incremental ...

On the Exploration of Incremental Learning for Fine-grained Image Retrieval

In this paper, we consider the problem of fine-grained image retrieval i...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.