Distilling Causal Effect of Data in Class-Incremental Learning

03/02/2021
by   Xinting Hu, et al.
1

We propose a causal framework to explain the catastrophic forgetting in Class-Incremental Learning (CIL) and then derive a novel distillation method that is orthogonal to the existing anti-forgetting techniques, such as data replay and feature/label distillation. We first 1) place CIL into the framework, 2) answer why the forgetting happens: the causal effect of the old data is lost in new training, and then 3) explain how the existing techniques mitigate it: they bring the causal effect back. Based on the framework, we find that although the feature/label distillation is storage-efficient, its causal effect is not coherent with the end-to-end feature learning merit, which is however preserved by data replay. To this end, we propose to distill the Colliding Effect between the old and the new data, which is fundamentally equivalent to the causal effect of data replay, but without any cost of replay storage. Thanks to the causal effect analysis, we can further capture the Incremental Momentum Effect of the data stream, removing which can help to retain the old effect overwhelmed by the new data effect, and thus alleviate the forgetting of the old class in testing. Extensive experiments on three CIL benchmarks: CIFAR-100, ImageNet-Sub Full, show that the proposed causal effect distillation can improve various state-of-the-art CIL methods by a large margin (0.72

READ FULL TEXT

page 1

page 2

page 3

page 4

06/17/2021

Dual-Teacher Class-Incremental Learning With Data-Free Generative Replay

This paper proposes two novel knowledge transfer techniques for class-in...
08/16/2021

Causal Incremental Graph Convolution for Recommender System Retraining

Real-world recommender system needs to be regularly retrained to keep wi...
07/05/2021

Multi-View Correlation Distillation for Incremental Object Detection

In real applications, new object classes often emerge after the detectio...
03/10/2022

Online Deep Metric Learning via Mutual Distillation

Deep metric learning aims to transform input data into an embedding spac...
05/25/2022

Semi-supervised Drifted Stream Learning with Short Lookback

In many scenarios, 1) data streams are generated in real time; 2) labele...
03/29/2019

Incremental Learning with Unlabeled Data in the Wild

Deep neural networks are known to suffer from catastrophic forgetting in...
05/06/2022

Forget Less, Count Better: A Domain-Incremental Self-Distillation Learning Benchmark for Lifelong Crowd Counting

Crowd Counting has important applications in public safety and pandemic ...