Scaling up learning with GAIT-prop

02/23/2021
by   Sander Dalm, et al.
0

Backpropagation of error (BP) is a widely used and highly successful learning algorithm. However, its reliance on non-local information in propagating error gradients makes it seem an unlikely candidate for learning in the brain. In the last decade, a number of investigations have been carried out focused upon determining whether alternative more biologically plausible computations can be used to approximate BP. This work builds on such a local learning algorithm - Gradient Adjusted Incremental Target Propagation (GAIT-prop) - which has recently been shown to approximate BP in a manner which appears biologically plausible. This method constructs local, layer-wise weight update targets in order to enable plausible credit assignment. However, in deep networks, the local weight updates computed by GAIT-prop can deviate from BP for a number of reasons. Here, we provide and test methods to overcome such sources of error. In particular, we adaptively rescale the locally-computed errors and show that this significantly increases the performance and stability of the GAIT-prop algorithm when applied to the CIFAR-10 dataset.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/11/2020

GAIT-prop: A biologically plausible learning rule derived from backpropagation of error

Traditional backpropagation of error, though a highly successful algorit...
research
10/16/2020

Towards truly local gradients with CLAPP: Contrastive, Local And Predictive Plasticity

Back-propagation (BP) is costly to implement in hardware and implausible...
research
01/31/2022

Towards Scaling Difference Target Propagation by Learning Backprop Targets

The development of biologically-plausible learning algorithms is importa...
research
05/29/2023

Understanding Predictive Coding as an Adaptive Trust-Region Method

Predictive coding (PC) is a brain-inspired local learning algorithm that...
research
09/23/2022

Hebbian Deep Learning Without Feedback

Recent approximations to backpropagation (BP) have mitigated many of BP'...
research
05/23/2023

Understanding and Improving Optimization in Predictive Coding Networks

Backpropagation (BP), the standard learning algorithm for artificial neu...
research
07/12/2018

Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures

The backpropagation of error algorithm (BP) is often said to be impossib...

Please sign up or login with your details

Forgot password? Click here to reset