IB-DRR: Incremental Learning with Information-Back Discrete Representation Replay

04/21/2021
by   Jian Jiang, et al.
15

Incremental learning aims to enable machine learning models to continuously acquire new knowledge given new classes, while maintaining the knowledge already learned for old classes. Saving a subset of training samples of previously seen classes in the memory and replaying them during new training phases is proven to be an efficient and effective way to fulfil this aim. It is evident that the larger number of exemplars the model inherits the better performance it can achieve. However, finding a trade-off between the model performance and the number of samples to save for each class is still an open problem for replay-based incremental learning and is increasingly desirable for real-life applications. In this paper, we approach this open problem by tapping into a two-step compression approach. The first step is a lossy compression, we propose to encode input images and save their discrete latent representations in the form of codes that are learned using a hierarchical Vector Quantised Variational Autoencoder (VQ-VAE). In the second step, we further compress codes losslessly by learning a hierarchical latent variable model with bits-back asymmetric numeral systems (BB-ANS). To compensate for the information lost in the first step compression, we introduce an Information Back (IB) mechanism that utilizes real exemplars for a contrastive learning loss to regularize the training of a classifier. By maintaining all seen exemplars' representations in the format of `codes', Discrete Representation Replay (DRR) outperforms the state-of-art method on CIFAR-100 by a margin of 4 memory cost required for saving samples. Incorporated with IB and saving a small set of old raw exemplars as well, the accuracy of DRR can be further improved by 2

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 5

page 11

page 12

06/17/2021

Dual-Teacher Class-Incremental Learning With Data-Free Generative Replay

This paper proposes two novel knowledge transfer techniques for class-in...
04/21/2021

Lossless Compression with Latent Variable Models

We develop a simple and elegant method for lossless compression using la...
03/30/2020

Incremental Learning In Online Scenario

Modern deep learning approaches have achieved great success in many visi...
03/29/2021

ClaRe: Practical Class Incremental Learning By Remembering Previous Class Representations

This paper presents a practical and simple yet efficient method to effec...
12/13/2020

Open-World Class Discovery with Kernel Networks

We study an Open-World Class Discovery problem in which, given labeled t...
03/31/2021

DER: Dynamically Expandable Representation for Class Incremental Learning

We address the problem of class incremental learning, which is a core st...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.