DeepAI
Log In Sign Up

Initial Classifier Weights Replay for Memoryless Class Incremental Learning

08/31/2020
by   Eden Belouadah, et al.
0

Incremental Learning (IL) is useful when artificial systems need to deal with streams of data and do not have access to all data at all times. The most challenging setting requires a constant complexity of the deep model and an incremental model update without access to a bounded memory of past data. Then, the representations of past classes are strongly affected by catastrophic forgetting. To mitigate its negative effect, an adapted fine tuning which includes knowledge distillation is usually deployed. We propose a different approach based on a vanilla fine tuning backbone. It leverages initial classifier weights which provide a strong representation of past classes because they are trained with all class data. However, the magnitude of classifiers learned in different states varies and normalization is needed for a fair handling of all classes. Normalization is performed by standardizing the initial classifier weights, which are assumed to be normally distributed. In addition, a calibration of prediction scores is done by using state level statistics to further improve classification fairness. We conduct a thorough evaluation with four public datasets in a memoryless incremental learning setting. Results show that our method outperforms existing techniques by a large margin for large-scale datasets.

READ FULL TEXT

page 1

page 2

page 4

01/16/2020

ScaIL: Classifier Weights Scaling for Class Incremental Learning

Incremental learning is useful if an AI agent needs to integrate data fr...
09/14/2022

PlaStIL: Plastic and Stable Memory-Free Class-Incremental Learning

Plasticity and stability are needed in class-incremental learning in ord...
11/23/2022

FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning

Exemplar-free class-incremental learning is very challenging due to the ...
02/01/2022

A Comparative Study of Calibration Methods for Imbalanced Class Incremental Learning

Deep learning approaches are successful in a wide range of AI problems a...
03/31/2020

A Simple Class Decision Balancing for Incremental Learning

Class incremental learning (CIL) problem, in which a learning agent cont...
04/03/2022

Revisiting a kNN-based Image Classification System with High-capacity Storage

In existing image classification systems that use deep neural networks, ...
09/01/2022

An Incremental Learning framework for Large-scale CTR Prediction

In this work we introduce an incremental learning framework for Click-Th...