Practical Riemannian Neural Networks

02/25/2016
by   Gaétan Marceau-Caron, et al.
0

We provide the first experimental results on non-synthetic datasets for the quasi-diagonal Riemannian gradient descents for neural networks introduced in [Ollivier, 2015]. These include the MNIST, SVHN, and FACE datasets as well as a previously unpublished electroencephalogram dataset. The quasi-diagonal Riemannian algorithms consistently beat simple stochastic gradient gradient descents by a varying margin. The computational overhead with respect to simple backpropagation is around a factor 2. Perhaps more interestingly, these methods also reach their final performance quickly, thus requiring fewer training epochs and a smaller total computation time. We also present an implementation guide to these Riemannian gradient descents for neural networks, showing how the quasi-diagonal versions can be implemented with minimal effort on top of existing routines which compute gradients.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/09/2023

Scalable Stochastic Gradient Riemannian Langevin Dynamics in Non-Diagonal Metrics

Stochastic-gradient sampling methods are often used to perform Bayesian ...
research
06/16/2020

Practical Quasi-Newton Methods for Training Deep Neural Networks

We consider the development of practical stochastic quasi-Newton, and in...
research
08/22/2018

Fisher Information and Natural Gradient Learning of Random Deep Networks

A deep neural network is a hierarchical nonlinear model transforming inp...
research
10/10/2022

Rieoptax: Riemannian Optimization in JAX

We present Rieoptax, an open source Python library for Riemannian optimi...
research
02/15/2023

Non-adiabatic Berry phase for semiconductor heavy holes under the coexistence of Rashba and Dresselhaus spin-orbit interactions

We formulate the non-Abelian Berry connection (tensor ℝ) and phase (matr...
research
05/14/2021

Efficient Quasi-Geodesics on the Stiefel Manifold

Solving the so-called geodesic endpoint problem, i.e., finding a geodesi...
research
07/16/2021

Controlling Recurrent Neural Networks by Diagonal Conceptors

The human brain is capable of learning, memorizing, and regenerating a p...

Please sign up or login with your details

Forgot password? Click here to reset