DeepAI AI Chat
Log In Sign Up

Reducing catastrophic forgetting with learning on synthetic data

by   Wojciech Masarczyk, et al.

Catastrophic forgetting is a problem caused by neural networks' inability to learn data in sequence. After learning two tasks in sequence, performance on the first one drops significantly. This is a serious disadvantage that prevents many deep learning applications to real-life problems where not all object classes are known beforehand; or change in data requires adjustments to the model. To reduce this problem we investigate the use of synthetic data, namely we answer a question: Is it possible to generate such data synthetically which learned in sequence does not result in catastrophic forgetting? We propose a method to generate such data in two-step optimisation process via meta-gradients. Our experimental results on Split-MNIST dataset show that training a model on such synthetic data in sequence does not result in catastrophic forgetting. We also show that our method of generating data is robust to different learning scenarios.


page 2

page 4


Catastrophic Importance of Catastrophic Forgetting

This paper describes some of the possibilities of artificial neural netw...

Attention-Based Structural-Plasticity

Catastrophic forgetting/interference is a critical problem for lifelong ...

Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives

In this paper, we investigate the instability in the standard dense retr...

Lifelong Learning with Searchable Extension Units

Lifelong learning remains an open problem. One of its main difficulties ...

Pseudo Rehearsal using non photo-realistic images

Deep Neural networks forget previously learnt tasks when they are faced ...

Combating catastrophic forgetting with developmental compression

Generally intelligent agents exhibit successful behavior across problems...

Dynamic memory to alleviate catastrophic forgetting in continuous learning settings

In medical imaging, technical progress or changes in diagnostic procedur...