Fortuitous Forgetting in Connectionist Networks

02/01/2022
by   Hattie Zhou, et al.
0

Forgetting is often seen as an unwanted characteristic in both human and machine learning. However, we propose that forgetting can in fact be favorable to learning. We introduce "forget-and-relearn" as a powerful paradigm for shaping the learning trajectories of artificial neural networks. In this process, the forgetting step selectively removes undesirable information from the model, and the relearning step reinforces features that are consistently useful under different conditions. The forget-and-relearn framework unifies many existing iterative training algorithms in the image classification and language emergence literature, and allows us to understand the success of these algorithms in terms of the disproportionate forgetting of undesirable information. We leverage this understanding to improve upon existing algorithms by designing more targeted forgetting operations. Insights from our analysis provide a coherent view on the dynamics of iterative training in neural networks and offer a clear path towards performance improvements.

READ FULL TEXT

page 22

page 23

page 24

research
11/21/2021

Learning by Active Forgetting for Neural Networks

Remembering and forgetting mechanisms are two sides of the same coin in ...
research
12/12/2018

An Empirical Study of Example Forgetting during Deep Neural Network Learning

Inspired by the phenomenon of catastrophic forgetting, we investigate th...
research
02/22/2018

Overcoming Catastrophic Forgetting in Convolutional Neural Networks by Selective Network Augmentation

Lifelong learning aims to develop machine learning systems that can lear...
research
05/25/2019

Lifelong Neural Predictive Coding: Sparsity Yields Less Forgetting when Learning Cumulatively

In lifelong learning systems, especially those based on artificial neura...
research
08/20/2019

Modelagem de um Problema de Dimensionamento de Lotes com Demanda Variavel e Deterministica e Efeitos de Learning e Forgetting

The main goal of this paper was to analyze the importance that the effec...
research
06/12/2020

Understanding the Role of Training Regimes in Continual Learning

Catastrophic forgetting affects the training of neural networks, limitin...
research
05/12/2023

Dual Forgetting Operators in the Context of Weakest Sufficient and Strongest Necessary Conditions

Forgetting is an important concept in knowledge representation and autom...

Please sign up or login with your details

Forgot password? Click here to reset