Sequence Learning using Equilibrium Propagation

by   Malyaban Bal, et al.

Equilibrium Propagation (EP) is a powerful and more bio-plausible alternative to conventional learning frameworks such as backpropagation. The effectiveness of EP stems from the fact that it relies only on local computations and requires solely one kind of computational unit during both of its training phases, thereby enabling greater applicability in domains such as bio-inspired neuromorphic computing. The dynamics of the model in EP is governed by an energy function and the internal states of the model consequently converge to a steady state following the state transition rules defined by the same. However, by definition, EP requires the input to the model (a convergent RNN) to be static in both the phases of training. Thus it is not possible to design a model for sequence classification using EP with an LSTM or GRU like architecture. In this paper, we leverage recent developments in modern hopfield networks to further understand energy based models and develop solutions for complex sequence classification tasks using EP while satisfying its convergence criteria and maintaining its theoretical similarities with recurrent backpropagation. We explore the possibility of integrating modern hopfield networks as an attention mechanism with convergent RNN models used in EP, thereby extending its applicability for the first time on two different sequence classification tasks in natural language processing viz. sentiment analysis (IMDB dataset) and natural language inference (SNLI dataset).


page 1

page 2

page 3

page 4


Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation

We introduce Equilibrium Propagation, a learning framework for energy-ba...

Recurrent Attention Unit

Recurrent Neural Network (RNN) has been successfully applied in many seq...

EqSpike: Spike-driven Equilibrium Propagation for Neuromorphic Implementations

Neuromorphic systems achieve high energy efficiency by computing with sp...

Holomorphic Equilibrium Propagation Computes Exact Gradients Through Finite Size Oscillations

Equilibrium propagation (EP) is an alternative to backpropagation (BP) t...

Equilibrium Propagation with Continual Weight Updates

Equilibrium Propagation (EP) is a learning algorithm that bridges Machin...

Updates of Equilibrium Prop Match Gradients of Backprop Through Time in an RNN with Static Input

Equilibrium Propagation (EP) is a biologically inspired learning algorit...

Please sign up or login with your details

Forgot password? Click here to reset