Meta-Learning Bidirectional Update Rules

by   Mark Sandler, et al.

In this paper, we introduce a new type of generalized neural network where neurons and synapses maintain multiple states. We show that classical gradient-based backpropagation in neural networks can be seen as a special case of a two-state network where one state is used for activations and another for gradients, with update rules derived from the chain rule. In our generalized framework, networks have neither explicit notion of nor ever receive gradients. The synapses and neurons are updated using a bidirectional Hebb-style update rule parameterized by a shared low-dimensional "genome". We show that such genomes can be meta-learned from scratch, using either conventional optimization techniques, or evolutionary strategies, such as CMA-ES. Resulting update rules generalize to unseen tasks and train faster than gradient descent based optimizers for several standard computer vision and synthetic tasks.


page 6

page 7

page 14


Finding online neural update rules by learning to remember

We investigate learning of the online local update rules for neural acti...

Meta-Learning Parameterized First-Order Optimizers using Differentiable Convex Optimization

Conventional optimization methods in machine learning and controls rely ...

Meta Learning Backpropagation And Improving It

Many concepts have been proposed for meta learning with neural networks ...

Meta-Learning with Warped Gradient Descent

A versatile and effective approach to meta-learning is to infer a gradie...

Learning Representations by Stochastic Meta-Gradient Descent in Neural Networks

Representations are fundamental to artificial intelligence. The performa...

Augmenting Supervised Learning by Meta-learning Unsupervised Local Rules

The brain performs unsupervised learning and (perhaps) simultaneous supe...

Generalizing Backpropagation for Gradient-Based Interpretability

Many popular feature-attribution methods for interpreting deep neural ne...

Please sign up or login with your details

Forgot password? Click here to reset