DeepAI AI Chat
Log In Sign Up

Saliency Guided Experience Packing for Replay in Continual Learning

by   Gobinda Saha, et al.
Purdue University

Artificial learning systems aspire to mimic human intelligence by continually learning from a stream of tasks without forgetting past knowledge. One way to enable such learning is to store past experiences in the form of input examples in episodic memory and replay them when learning new tasks. However, performance of such method suffers as the size of the memory becomes smaller. In this paper, we propose a new approach for experience replay, where we select the past experiences by looking at the saliency maps which provide visual explanations for the model's decision. Guided by these saliency maps, we pack the memory with only the parts or patches of the input images important for the model's prediction. While learning a new task, we replay these memory patches with appropriate zero-padding to remind the model about its past decisions. We evaluate our algorithm on diverse image classification datasets and report better performance than the state-of-the-art approaches. With qualitative and quantitative analyses we show that our method captures richer summary of past experiences without any memory increase, and hence performs well with small episodic memory.


page 3

page 6

page 7


Gradient Projection Memory for Continual Learning

The ability to learn continually without forgetting the past tasks is a ...

Saliency-Augmented Memory Completion for Continual Learning

Continual Learning is considered a key step toward next-generation Artif...

Revisiting Experience Replay: Continual Learning by Adaptively Tuning Task-wise Relationship

Continual learning requires models to learn new tasks while maintaining ...

SYNERgy between SYNaptic consolidation and Experience Replay for general continual learning

Continual learning (CL) in the brain is facilitated by a complex set of ...

The Effectiveness of Memory Replay in Large Scale Continual Learning

We study continual learning in the large scale setting where tasks in th...

Episodic Memory in Lifelong Language Learning

We introduce a lifelong language learning setup where a model needs to l...

Continual Learning with Self-Organizing Maps

Despite remarkable successes achieved by modern neural networks in a wid...