Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias

by   Axel Laborieux, et al.

Equilibrium Propagation (EP) is a biologically-inspired algorithm for convergent RNNs with a local learning rule that comes with strong theoretical guarantees. The parameter updates of the neural network during the credit assignment phase have been shown mathematically to approach the gradients provided by Backpropagation Through Time (BPTT) when the network is infinitesimally nudged toward its target. In practice, however, training a network with the gradient estimates provided by EP does not scale to visual tasks harder than MNIST. In this work, we show that a bias in the gradient estimate of EP, inherent in the use of finite nudging, is responsible for this phenomenon and that cancelling it allows training deep ConvNets by EP. We show that this bias can be greatly reduced by using symmetric nudging (a positive nudging and a negative one). We also generalize previous EP equations to the case of cross-entropy loss (by opposition to squared error). As a result of these advances, we are able to achieve a test error of 11.7 which approaches the one achieved by BPTT and provides a major improvement with respect to the standard EP approach with same-sign nudging that gives 86 error. We also apply these techniques to train an architecture with asymmetric forward and backward connections, yielding a 13.2 highlight EP as a compelling biologically-plausible approach to compute error gradients in deep neural networks.


page 1

page 2

page 3

page 4


The HSIC Bottleneck: Deep Learning without Back-Propagation

We introduce the HSIC (Hilbert-Schmidt independence criterion) bottlenec...

Updates of Equilibrium Prop Match Gradients of Backprop Through Time in an RNN with Static Input

Equilibrium Propagation (EP) is a biologically inspired learning algorit...

Continual Weight Updates and Convolutional Architectures for Equilibrium Propagation

Equilibrium Propagation (EP) is a biologically inspired alternative algo...

Scaling Forward Gradient With Local Losses

Forward gradient learning computes a noisy directional gradient and is a...

Estimator Meets Equilibrium Perspective: A Rectified Straight Through Estimator for Binary Neural Networks Training

Binarization of neural networks is a dominant paradigm in neural network...

Sign and Relevance learning

Standard models of biologically realistic, or inspired, reinforcement le...

Equilibrium Propagation with Continual Weight Updates

Equilibrium Propagation (EP) is a learning algorithm that bridges Machin...

Please sign up or login with your details

Forgot password? Click here to reset