Training End-to-End Analog Neural Networks with Equilibrium Propagation

06/02/2020
by   Jack Kendall, et al.
9

We introduce a principled method to train end-to-end analog neural networks by stochastic gradient descent. In these analog neural networks, the weights to be adjusted are implemented by the conductances of programmable resistive devices such as memristors [Chua, 1971], and the nonlinear transfer functions (or `activation functions') are implemented by nonlinear components such as diodes. We show mathematically that a class of analog neural networks (called nonlinear resistive networks) are energy-based models: they possess an energy function as a consequence of Kirchhoff's laws governing electrical circuits. This property enables us to train them using the Equilibrium Propagation framework [Scellier and Bengio, 2017]. Our update rule for each conductance, which is local and relies solely on the voltage drop across the corresponding resistor, is shown to compute the gradient of the loss function. Our numerical simulations, which use the SPICE-based Spectre simulation framework to simulate the dynamics of electrical circuits, demonstrate training on the MNIST classification task, performing comparably or better than equivalent-size software-based neural networks. Our work can guide the development of a new generation of ultra-fast, compact and low-power neural networks supporting on-chip learning.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/08/2023

Implementation Of MNIST Dataset Learning Using Analog Circuit

There have been many attempts to implement neural networks in the analog...
research
11/25/2018

On-chip learning for domain wall synapse based Fully Connected Neural Network

Spintronic devices are considered as promising candidates in implementin...
research
03/12/2019

Reprogrammable Electro-Optic Nonlinear Activation Functions for Optical Neural Networks

We introduce an electro-optic hardware platform for nonlinear activation...
research
03/25/2021

Training Neural Networks Using the Property of Negative Feedback to Inverse a Function

With high forward gain, a negative feedback system has the ability to pe...
research
08/10/2022

Frequency propagation: Multi-mechanism learning in nonlinear physical networks

We introduce frequency propagation, a learning algorithm for nonlinear p...
research
12/16/2017

Mitigating Asymmetric Nonlinear Weight Update Effects in Hardware Neural Network based on Analog Resistive Synapse

Asymmetric nonlinear weight update is considered as one of the major obs...
research
10/15/2019

The Local Elasticity of Neural Networks

This paper presents a phenomenon in neural networks that we refer to as ...

Please sign up or login with your details

Forgot password? Click here to reset