DeepAI AI Chat
Log In Sign Up

Topological obstructions in neural networks learning

12/31/2020
by   Serguei Barannikov, et al.
0

We apply methods of topological data analysis to loss functions to gain insights on learning of deep neural networks and their generalization properties. We study global properties of the loss function gradient flow. We use topological data analysis of the loss function and its Morse complex to relate local behavior along gradient trajectories with global properties of the loss surface. We define neural network Topological Obstructions score, TO-score, with help of robust topological invariants, barcodes of loss function, that quantify the badness of local minima for gradient-based optimization. We have made several experiments for computing these invariants, for small neural networks, and for fully connected, convolutional and ResNet-like neural networks on different datasets: MNIST, Fashion MNIST, CIFAR10, SVHN. Our two principal observations are as follows. Firstly, the neural network barcode and TO-score decrease with the increase of the neural network depth and width. Secondly, there is an intriguing connection between the length of minima segments in the barcode and the minima generalization error.

READ FULL TEXT

page 14

page 15

11/29/2019

Barcodes as summary of objective function's topology

We apply the canonical forms (barcodes) of gradient Morse complexes to e...
10/30/2017

The loss surface and expressivity of deep convolutional neural networks

We analyze the expressiveness and loss surface of practical deep convolu...
05/26/2018

Deep Learning Topological Invariants of Band Insulators

In this work we design and train deep neural networks to predict topolog...
01/08/2019

Visualising Basins of Attraction for the Cross-Entropy and the Squared Error Neural Network Loss Functions

Quantification of the stationary points and the associated basins of att...
04/12/2022

Local and global topological complexity measures OF ReLU neural network functions

We apply a generalized piecewise-linear (PL) version of Morse theory due...
05/24/2022

Functional Network: A Novel Framework for Interpretability of Deep Neural Networks

The layered structure of deep neural networks hinders the use of numerou...
03/06/2023

Finding metastable skyrmionic structures via a metaheuristic perturbation-driven neural network

Topological magnetic textures observed in experiments can, in principle,...