Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting

by   Xialei Liu, et al.
Universitat Autònoma de Barcelona

In this paper we propose an approach to avoiding catastrophic forgetting in sequential task learning scenarios. Our technique is based on a network reparameterization that approximately diagonalizes the Fisher Information Matrix of the network parameters. This reparameterization takes the form of a factorized rotation of parameter space which, when used in conjunction with Elastic Weight Consolidation (which assumes a diagonal Fisher Information Matrix), leads to significantly better performance on lifelong learning of sequential tasks. Experimental results on the MNIST, CIFAR-100, CUB-200 and Stanford-40 datasets demonstrate that we significantly improve the results of standard elastic weight consolidation, and that we obtain competitive results when compared to other state-of-the-art in lifelong learning without forgetting.


Towards continuous learning for glioma segmentation with elastic weight consolidation

When finetuning a convolutional neural network (CNN) on data from a new ...

Lifelong Learning with Sketched Structural Regularization

Preventing catastrophic forgetting while continually learning new tasks ...

Kronecker Factorization for Preventing Catastrophic Forgetting in Large-scale Medical Entity Linking

Multi-task learning is useful in NLP because it is often practically des...

Overcoming catastrophic forgetting problem by weight consolidation and long-term memory

Sequential learning of multiple tasks in artificial neural networks usin...

Beneficial perturbation network for continual learning

Sequential learning of multiple tasks in artificial neural networks usin...

Sparsifying networks by traversing Geodesics

The geometry of weight spaces and functional manifolds of neural network...

Monitoring nonstationary processes based on recursive cointegration analysis and elastic weight consolidation

This paper considers the problem of nonstationary process monitoring und...

Code Repositories


The rep for the RotateNetworks in ICPR18, Beijing, China.

view repo

Please sign up or login with your details

Forgot password? Click here to reset