Meta Learning Backpropagation And Improving It

12/29/2020
by   Louis Kirsch, et al.
92

Many concepts have been proposed for meta learning with neural networks (NNs), e.g., NNs that learn to control fast weights, hyper networks, learned learning rules, and meta recurrent NNs. Our Variable Shared Meta Learning (VS-ML) unifies the above and demonstrates that simple weight-sharing and sparsity in an NN is sufficient to express powerful learning algorithms (LAs) in a reusable fashion. A simple implementation of VS-ML called VS-ML RNN allows for implementing the backpropagation LA solely by running an RNN in forward-mode. It can even meta-learn new LAs that improve upon backpropagation and generalize to datasets outside of the meta training distribution without explicit gradient calculation. Introspection reveals that our meta-learned LAs learn qualitatively different from gradient descent through fast association.

READ FULL TEXT

page 3

page 7

research
08/30/2019

Meta-Learning with Warped Gradient Descent

A versatile and effective approach to meta-learning is to infer a gradie...
research
10/27/2021

Learning where to learn: Gradient sparsity in meta and continual learning

Finding neural network weights that generalize well from small datasets ...
research
10/31/2017

Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm

Learning to learn is a powerful paradigm for enabling models to learn fr...
research
04/10/2021

Meta-Learning Bidirectional Update Rules

In this paper, we introduce a new type of generalized neural network whe...
research
02/08/2020

ML-misfit: Learning a robust misfit function for full-waveform inversion using machine learning

Most of the available advanced misfit functions for full waveform invers...
research
02/11/2022

A Modern Self-Referential Weight Matrix That Learns to Modify Itself

The weight matrix (WM) of a neural network (NN) is its program. The prog...
research
07/04/2022

The least-control principle for learning at equilibrium

Equilibrium systems are a powerful way to express neural computations. A...

Please sign up or login with your details

Forgot password? Click here to reset