The Local Elasticity of Neural Networks

10/15/2019
by   Hangfeng He, et al.
22

This paper presents a phenomenon in neural networks that we refer to as local elasticity. Roughly speaking, a classifier is said to be locally elastic if its prediction at a feature vector ' is not significantly perturbed, after the classifier is updated via stochastic gradient descent at a (labeled) feature vector that is dissimilar to ' in a certain sense. This phenomenon is shown to persist for neural networks with nonlinear activation functions through extensive simulations on real-life and synthetic datasets, whereas this is not observed in linear classifiers. In addition, we offer a geometric interpretation of local elasticity using the neural tangent kernel <cit.>. Building on top of local elasticity, we obtain pairwise similarity measures between feature vectors, which can be used for clustering in conjunction with K-means. The effectiveness of the clustering algorithm on the MNIST and CIFAR-10 datasets in turn corroborates the hypothesis of local elasticity of neural networks on real-life data. Finally, we discuss some implications of local elasticity to shed light on several intriguing aspects of deep neural networks.

READ FULL TEXT
research
10/11/2021

Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations

Understanding the training dynamics of deep learning models is perhaps a...
research
09/25/2019

Locally adaptive activation functions with slope recovery term for deep and physics-informed neural networks

We propose two approaches of locally adaptive activation functions namel...
research
10/27/2020

Toward Better Generalization Bounds with Locally Elastic Stability

Classical approaches in learning theory are often seen to yield very loo...
research
10/05/2014

Understanding Locally Competitive Networks

Recently proposed neural network activation functions such as rectified ...
research
06/02/2020

Training End-to-End Analog Neural Networks with Equilibrium Propagation

We introduce a principled method to train end-to-end analog neural netwo...

Please sign up or login with your details

Forgot password? Click here to reset