Weakly-correlated synapses promote dimension reduction in deep neural networks

06/20/2020
by   Jianwen Zhou, et al.
0

By controlling synaptic and neural correlations, deep learning has achieved empirical successes in improving classification performances. How synaptic correlations affect neural correlations to produce disentangled hidden representations remains elusive. Here we propose a simplified model of dimension reduction, taking into account pairwise correlations among synapses, to reveal the mechanism underlying how the synaptic correlations affect dimension reduction. Our theory determines the synaptic-correlation scaling form requiring only mathematical self-consistency, for both binary and continuous synapses. The theory also predicts that weakly-correlated synapses encourage dimension reduction compared to their orthogonal counterparts. In addition, these synapses slow down the decorrelation process along the network depth. These two computational roles are explained by the proposed mean-field equation. The theoretical predictions are in excellent agreement with numerical simulations, and the key features are also captured by a deep learning with Hebbian rules.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/07/2018

Metric dimension reduction: A snapshot of the Ribe program

The purpose of this article is to survey some of the context, achievemen...
research
10/30/2017

Time-lagged autoencoders: Deep learning of slow collective variables for molecular kinetics

Inspired by the success of deep learning techniques in the physical and ...
research
11/07/2012

Randomized Dimension Reduction on Massive Data

Scalability of statistical estimators is of increasing importance in mod...
research
10/28/2020

Bridging linearity-based and kernel-based sufficient dimension reduction

There has been a lot of interest in sufficient dimension reduction (SDR)...
research
06/20/2016

Continuum directions for supervised dimension reduction

Dimension reduction of multivariate data supervised by auxiliary informa...
research
11/27/2022

Linear Classification of Neural Manifolds with Correlated Variability

Understanding how the statistical and geometric properties of neural act...
research
07/02/2022

Pavlov Learning Machines

As well known, Hebb's learning traces its origin in Pavlov's Classical C...

Please sign up or login with your details

Forgot password? Click here to reset