Improving equilibrium propagation without weight symmetry through Jacobian homeostasis

by   Axel Laborieux, et al.

Equilibrium propagation (EP) is a compelling alternative to the backpropagation of error algorithm (BP) for computing gradients of neural networks on biological or analog neuromorphic substrates. Still, the algorithm requires weight symmetry and infinitesimal equilibrium perturbations, i.e., nudges, to estimate unbiased gradients efficiently. Both requirements are challenging to implement in physical systems. Yet, whether and how weight asymmetry affects its applicability is unknown because, in practice, it may be masked by biases introduced through the finite nudge. To address this question, we study generalized EP, which can be formulated without weight symmetry, and analytically isolate the two sources of bias. For complex-differentiable non-symmetric networks, we show that the finite nudge does not pose a problem, as exact derivatives can still be estimated via a Cauchy integral. In contrast, weight asymmetry introduces bias resulting in low task performance due to poor alignment of EP's neuronal error vectors compared to BP. To mitigate this issue, we present a new homeostatic objective that directly penalizes functional asymmetries of the Jacobian at the network's fixed point. This homeostatic objective dramatically improves the network's ability to solve complex tasks such as ImageNet 32x32. Our results lay the theoretical groundwork for studying and mitigating the adverse effects of imperfections of physical networks on learning algorithms that rely on the substrate's relaxation dynamics.


page 1

page 2

page 3

page 4


Holomorphic Equilibrium Propagation Computes Exact Gradients Through Finite Size Oscillations

Equilibrium propagation (EP) is an alternative to backpropagation (BP) t...

Biologically-plausible learning algorithms can scale to large datasets

The backpropagation (BP) algorithm is often thought to be biologically i...

Equivalence of Equilibrium Propagation and Recurrent Backpropagation

Recurrent Backpropagation and Equilibrium Propagation are algorithms for...

Generalization of Equilibrium Propagation to Vector Field Dynamics

The biological plausibility of the backpropagation algorithm has long be...

Learning with augmented target information: An alternative theory of Feedback Alignment

While error backpropagation (BP) has dominated the training of nearly al...

Continual Weight Updates and Convolutional Architectures for Equilibrium Propagation

Equilibrium Propagation (EP) is a biologically inspired alternative algo...

Learning by non-interfering feedback chemical signaling in physical networks

Both non-neural and neural biological systems can learn. So rather than ...

Please sign up or login with your details

Forgot password? Click here to reset