On the training of sparse and dense deep neural networks: less parameters, same performance

06/17/2021
by   Lorenzo Chicchi, et al.
0

Deep neural networks can be trained in reciprocal space, by acting on the eigenvalues and eigenvectors of suitable transfer operators in direct space. Adjusting the eigenvalues, while freezing the eigenvectors, yields a substantial compression of the parameter space. This latter scales by definition with the number of computing neurons. The classification scores, as measured by the displayed accuracy, are however inferior to those attained when the learning is carried in direct space, for an identical architecture and by employing the full set of trainable parameters (with a quadratic dependence on the size of neighbor layers). In this Letter, we propose a variant of the spectral learning method as appeared in Giambagli et al Nat. Comm. 2021, which leverages on two sets of eigenvalues, for each mapping between adjacent layers. The eigenvalues act as veritable knobs which can be freely tuned so as to (i) enhance, or alternatively silence, the contribution of the input nodes, (ii) modulate the excitability of the receiving nodes with a mechanism which we interpret as the artificial analogue of the homeostatic plasticity. The number of trainable parameters is still a linear function of the network size, but the performances of the trained device gets much closer to those obtained via conventional algorithms, these latter requiring however a considerably heavier computational cost. The residual gap between conventional and spectral trainings can be eventually filled by employing a suitable decomposition for the non trivial block of the eigenvectors matrix. Each spectral parameter reflects back on the whole set of inter-nodes weights, an attribute which we shall effectively exploit to yield sparse networks with stunning classification abilities, as compared to their homologues trained with conventional means.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

05/29/2020

Machine learning in spectral domain

Deep neural networks are usually trained in the space of the nodes, by a...
06/15/2020

Finding trainable sparse networks through Neural Tangent Transfer

Deep neural networks have dramatically transformed machine learning, but...
01/07/2022

Block Walsh-Hadamard Transform Based Binary Layers in Deep Neural Networks

Convolution has been the core operation of modern deep neural networks. ...
11/25/2019

Rigging the Lottery: Making All Tickets Winners

Sparse neural networks have been shown to be more parameter and compute ...
05/01/2019

High-performance sampling of generic Determinantal Point Processes

Determinantal Point Processes (DPPs) were introduced by Macchi as a mode...
11/30/2021

Approximate Spectral Decomposition of Fisher Information Matrix for Simple ReLU Networks

We investigate the Fisher information matrix (FIM) of one hidden layer n...
09/25/2019

Stochastic collocation method for computing eigenspaces of parameter-dependent operators

We consider computing eigenspaces of an elliptic self-adjoint operator d...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.