DeepAI AI Chat
Log In Sign Up

NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks

by   Mustafa Burak Gurbuz, et al.

The goal of continual learning (CL) is to learn different tasks over time. The main desiderata associated with CL are to maintain performance on older tasks, leverage the latter to improve learning of future tasks, and to introduce minimal overhead in the training process (for instance, to not require a growing model or retraining). We propose the Neuro-Inspired Stability-Plasticity Adaptation (NISPA) architecture that addresses these desiderata through a sparse neural network with fixed density. NISPA forms stable paths to preserve learned knowledge from older tasks. Also, NISPA uses connection rewiring to create new plastic paths that reuse existing knowledge on novel tasks. Our extensive evaluation on EMNIST, FashionMNIST, CIFAR10, and CIFAR100 datasets shows that NISPA significantly outperforms representative state-of-the-art continual learning baselines, and it uses up to ten times fewer learnable parameters compared to baselines. We also make the case that sparsity is an essential ingredient for continual learning. The NISPA code is available at


page 7

page 8


Bilevel Continual Learning

Continual learning aims to learn continuously from a stream of tasks and...

TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models

Language Models (LMs) become outdated as the world changes; they often f...

Continual Learning for Text Classification with Information Disentanglement Based Regularization

Continual learning has become increasingly important as it enables NLP m...

Compression-aware Continual Learning using Singular Value Decomposition

We propose a compression based continual task learning method that can d...

Exploring Example Influence in Continual Learning

Continual Learning (CL) sequentially learns new tasks like human beings,...

Meta-attention for ViT-backed Continual Learning

Continual learning is a longstanding research topic due to its crucial r...

Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning

Models trained in the context of continual learning (CL) should be able ...