Natural Neural Networks

07/01/2015
by   Guillaume Desjardins, et al.
0

We introduce Natural Neural Networks, a novel family of algorithms that speed up convergence by adapting their internal representation during training to improve conditioning of the Fisher matrix. In particular, we show a specific example that employs a simple and efficient reparametrization of the neural network weights by implicitly whitening the representation obtained at each layer, while preserving the feed-forward computation of the network. Such networks can be trained efficiently via the proposed Projected Natural Gradient Descent algorithm (PRONG), which amortizes the cost of these reparametrizations over many parameter updates and is closely related to the Mirror Descent online learning algorithm. We highlight the benefits of our method on both unsupervised and supervised learning tasks, and showcase its scalability by training on the large-scale ImageNet Challenge dataset.

READ FULL TEXT
research
12/11/2019

A Supervised Modified Hebbian Learning Method On Feed-forward Neural Networks

In this paper, we present a new supervised learning algorithm that is ba...
research
03/04/2013

Riemannian metrics for neural networks I: feedforward networks

We describe four algorithms for neural network training, each adapted to...
research
04/24/2017

A Neural Network model with Bidirectional Whitening

We present here a new model and algorithm which performs an efficient Na...
research
05/10/2018

Monotone Learning with Rectifier Networks

We introduce a new neural network model, together with a tractable and m...
research
05/10/2018

Monotone Learning with Rectified Wire Networks

We introduce a new neural network model, together with a tractable and m...
research
08/06/2019

Refining the Structure of Neural Networks Using Matrix Conditioning

Deep learning models have proven to be exceptionally useful in performin...
research
09/26/2018

Learning Preconditioners on Lie Groups

We study two types of preconditioners and preconditioned stochastic grad...

Please sign up or login with your details

Forgot password? Click here to reset