Continual Learning via Neural Pruning

03/11/2019
by   Siavash Golkar, et al.
14

We introduce Continual Learning via Neural Pruning (CLNP), a new method aimed at lifelong learning in fixed capacity models based on neuronal model sparsification. In this method, subsequent tasks are trained using the inactive neurons and filters of the sparsified network and cause zero deterioration to the performance of previous tasks. In order to deal with the possible compromise between model sparsity and performance, we formalize and incorporate the concept of graceful forgetting: the idea that it is preferable to suffer a small amount of forgetting in a controlled manner if it helps regain network capacity and prevents uncontrolled loss of performance during the training of future tasks. CLNP also provides simple continual learning diagnostic tools in terms of the number of free neurons left for the training of future tasks as well as the number of neurons that are being reused. In particular, we see in experiments that CLNP verifies and automatically takes advantage of the fact that the features of earlier layers are more transferable. We show empirically that CLNP leads to significantly improved results over current weight elasticity based methods.

READ FULL TEXT

page 8

page 9

research
02/21/2022

Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning

Continual Learning (CL) methods aim to enable machine learning models to...
research
10/15/2019

Compacting, Picking and Growing for Unforgetting Continual Learning

Continual lifelong learning is essential to many applications. In this p...
research
06/14/2018

Selfless Sequential Learning

Sequential learning studies the problem of learning tasks in a sequence ...
research
10/28/2020

A Study on Efficiency in Continual Learning Inspired by Human Learning

Humans are efficient continual learning systems; we continually learn ne...
research
07/17/2021

Continual Learning for Task-oriented Dialogue System with Iterative Network Pruning, Expanding and Masking

This ability to learn consecutive tasks without forgetting how to perfor...
research
06/02/2023

GateON: an unsupervised method for large scale continual learning

The objective of continual learning (CL) is to learn tasks sequentially ...
research
04/21/2020

Bayesian Nonparametric Weight Factorization for Continual Learning

Naively trained neural networks tend to experience catastrophic forgetti...

Please sign up or login with your details

Forgot password? Click here to reset