Continual Normalization: Rethinking Batch Normalization for Online Continual Learning

03/30/2022
by   Quang Pham, et al.
0

Existing continual learning methods use Batch Normalization (BN) to facilitate training and improve generalization across tasks. However, the non-i.i.d and non-stationary nature of continual learning data, especially in the online setting, amplify the discrepancy between training and testing in BN and hinder the performance of older tasks. In this work, we study the cross-task normalization effect of BN in online continual learning where BN normalizes the testing data using moments biased towards the current task, resulting in higher catastrophic forgetting. This limitation motivates us to propose a simple yet effective method that we call Continual Normalization (CN) to facilitate training similar to BN while mitigating its negative effect. Extensive experiments on different continual learning algorithms and online scenarios show that CN is a direct replacement for BN and can provide substantial performance improvements. Our implementation is available at <https://github.com/phquang/Continual-Normalization>.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/30/2020

Bilevel Continual Learning

Continual learning aims to learn continuously from a stream of tasks and...
research
06/29/2023

Improving Online Continual Learning Performance and Stability with Temporal Ensembles

Neural networks are very effective when trained on large datasets for a ...
research
08/20/2021

Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data

Continual learning is the problem of learning and retaining knowledge th...
research
01/29/2022

Task-Balanced Batch Normalization for Exemplar-based Class-Incremental Learning

Batch Normalization (BN) is an essential layer for training neural netwo...
research
09/11/2021

Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers

This paper investigates continual learning for semantic parsing. In this...
research
05/16/2023

Online Continual Learning Without the Storage Constraint

Online continual learning (OCL) research has primarily focused on mitiga...
research
07/28/2021

Task-Specific Normalization for Continual Learning of Blind Image Quality Models

The computational vision community has recently paid attention to contin...

Please sign up or login with your details

Forgot password? Click here to reset