Memory-based Parameter Adaptation

by   Pablo Sprechmann, et al.

Deep neural networks have excelled on a wide range of problems, from vision to language and game playing. Neural networks very gradually incorporate information into weights as they process data, requiring very low learning rates. If the training distribution shifts, the network is slow to adapt, and when it does adapt, it typically performs badly on the training distribution before the shift. Our method, Memory-based Parameter Adaptation, stores examples in memory and then uses a context-based lookup to directly modify the weights of a neural network. Much higher learning rates can be used for this local adaptation, reneging the need for many iterations over similar data before good predictions can be made. As our method is memory-based, it alleviates several shortcomings of neural networks, such as catastrophic forgetting, fast, stable acquisition of new knowledge, learning with an imbalanced class labels, and fast learning during evaluation. We demonstrate this on a range of supervised tasks: large-scale image classification and language modelling.


Representation Memorization for Fast Learning New Knowledge without Forgetting

The ability to quickly learn new knowledge (e.g. new classes or data dis...

Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling

Training a deep neural network requires a large amount of single-task da...

Revisiting Parameter Reuse to Overcome Catastrophic Forgetting in Neural Networks

Neural networks tend to forget previously learned knowledge when continu...

Encoding Hierarchical Information in Neural Networks helps in Subpopulation Shift

Over the past decade, deep neural networks have proven to be adept in im...

Rapid Adaptation with Conditionally Shifted Neurons

We describe a mechanism by which artificial neural networks can learn ra...

Fast Parametric Learning with Activation Memorization

Neural networks trained with backpropagation often struggle to identify ...

Editable Neural Networks

These days deep neural networks are ubiquitously used in a wide range of...

Code Repositories


Random memory adaptation model inspired by the paper: "Memory-based parameter adaptation (MbPA)"

view repo

Please sign up or login with your details

Forgot password? Click here to reset