Surrogate Gradients Design

02/01/2022
by   Luca Herranz-Celotti, et al.
0

Surrogate gradient (SG) training provides the possibility to quickly transfer all the gains made in deep learning to neuromorphic computing and neuromorphic processors, with the consequent reduction in energy consumption. Evidence supports that training can be robust to the choice of SG shape, after an extensive search of hyper-parameters. However, random or grid search of hyper-parameters becomes exponentially unfeasible as we consider more hyper-parameters. Moreover, every point in the search can itself be highly time and energy consuming for large networks and large datasets. In this article we show how complex tasks and networks are more sensitive to SG choice. Secondly, we show how low dampening, high sharpness and low tail fatness are preferred. Thirdly, we observe that Glorot Uniform initialization is generally preferred by most SG choices, with variability in the results. We finally provide a theoretical solution to reduce the need of extensive gridsearch, to find SG shape and initializations that result in improved accuracy.

READ FULL TEXT

page 8

page 9

research
09/25/2019

A Heuristic for Efficient Reduction in Hidden Layer Combinations For Feedforward Neural Networks

In this paper, we describe the hyper-parameter search problem in the fie...
research
11/06/2021

What augmentations are sensitive to hyper-parameters and why?

We apply augmentations to our dataset to enhance the quality of our pred...
research
06/13/2020

Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies

Off-policy learning algorithms have been known to be sensitive to the ch...
research
11/14/2022

An online algorithm for contrastive Principal Component Analysis

Finding informative low-dimensional representations that can be computed...
research
03/26/2022

A Novel Neuromorphic Processors Realization of Spiking Deep Reinforcement Learning for Portfolio Management

The process of continuously reallocating funds into financial assets, ai...
research
06/03/2021

Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks

Neuromorphic sensing and computing hold a promise for highly energy-effi...
research
07/07/2017

A case study of Empirical Bayes in User-Movie Recommendation system

In this article we provide a formulation of empirical bayes described by...

Please sign up or login with your details

Forgot password? Click here to reset