Schematic Memory Persistence and Transience for Efficient and Robust Continual Learning

05/05/2021
by   Yuyang Gao, et al.
10

Continual learning is considered a promising step towards next-generation Artificial Intelligence (AI), where deep neural networks (DNNs) make decisions by continuously learning a sequence of different tasks akin to human learning processes. It is still quite primitive, with existing works focusing primarily on avoiding (catastrophic) forgetting. However, since forgetting is inevitable given bounded memory and unbounded task loads, 'how to reasonably forget' is a problem continual learning must address in order to reduce the performance gap between AIs and humans, in terms of 1) memory efficiency, 2) generalizability, and 3) robustness when dealing with noisy data. To address this, we propose a novel ScheMAtic memory peRsistence and Transience (SMART) framework for continual learning with external memory that builds on recent advances in neuroscience. The efficiency and generalizability are enhanced by a novel long-term forgetting mechanism and schematic memory, using sparsity and 'backward positive transfer' constraints with theoretical guarantees on the error bound. Robust enhancement is achieved using a novel short-term forgetting mechanism inspired by background information-gated learning. Finally, an extensive experimental analysis on both benchmark and real-world datasets demonstrates the effectiveness and efficiency of our model.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/26/2022

Saliency-Augmented Memory Completion for Continual Learning

Continual Learning is considered a key step toward next-generation Artif...
research
03/26/2022

Continual learning of quantum state classification with gradient episodic memory

Continual learning is one of the many areas of machine learning research...
research
02/19/2021

Condensed Composite Memory Continual Learning

Deep Neural Networks (DNNs) suffer from a rapid decrease in performance ...
research
08/29/2023

Incorporating Neuro-Inspired Adaptability for Continual Learning in Artificial Intelligence

Continual learning aims to empower artificial intelligence (AI) with str...
research
11/23/2021

Reviewing continual learning from the perspective of human-level intelligence

Humans' continual learning (CL) ability is closely related to Stability ...
research
11/30/2021

Hierarchical Prototype Networks for Continual Graph Representation Learning

Despite significant advances in graph representation learning, little at...
research
02/12/2023

Theory on Forgetting and Generalization of Continual Learning

Continual learning (CL), which aims to learn a sequence of tasks, has at...

Please sign up or login with your details

Forgot password? Click here to reset