Memory-based Parameter Adaptation

02/28/2018
by   Pablo Sprechmann, et al.
0

Deep neural networks have excelled on a wide range of problems, from vision to language and game playing. Neural networks very gradually incorporate information into weights as they process data, requiring very low learning rates. If the training distribution shifts, the network is slow to adapt, and when it does adapt, it typically performs badly on the training distribution before the shift. Our method, Memory-based Parameter Adaptation, stores examples in memory and then uses a context-based lookup to directly modify the weights of a neural network. Much higher learning rates can be used for this local adaptation, reneging the need for many iterations over similar data before good predictions can be made. As our method is memory-based, it alleviates several shortcomings of neural networks, such as catastrophic forgetting, fast, stable acquisition of new knowledge, learning with an imbalanced class labels, and fast learning during evaluation. We demonstrate this on a range of supervised tasks: large-scale image classification and language modelling.

READ FULL TEXT
research
08/28/2021

Representation Memorization for Fast Learning New Knowledge without Forgetting

The ability to quickly learn new knowledge (e.g. new classes or data dis...
research
09/03/2020

Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling

Training a deep neural network requires a large amount of single-task da...
research
07/22/2022

Revisiting Parameter Reuse to Overcome Catastrophic Forgetting in Neural Networks

Neural networks tend to forget previously learned knowledge when continu...
research
12/20/2021

Encoding Hierarchical Information in Neural Networks helps in Subpopulation Shift

Over the past decade, deep neural networks have proven to be adept in im...
research
12/28/2017

Rapid Adaptation with Conditionally Shifted Neurons

We describe a mechanism by which artificial neural networks can learn ra...
research
03/27/2018

Fast Parametric Learning with Activation Memorization

Neural networks trained with backpropagation often struggle to identify ...
research
04/01/2020

Editable Neural Networks

These days deep neural networks are ubiquitously used in a wide range of...

Please sign up or login with your details

Forgot password? Click here to reset